[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:07:48 dist-test-slave-wd30 krb5kdc[819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:07:48 dist-test-slave-wd30 krb5kdc[819](info): set up 2 sockets
Jun 20 19:07:48 dist-test-slave-wd30 krb5kdc[819](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 19:07:49.583663 812 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:07:49 dist-test-slave-wd30 krb5kdc[819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:07:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:07:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:07:49.706362 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:36401
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:36401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:49.789784 832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:49.790051 832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:49.790088 832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:49.791370 832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:07:49.791435 832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:49.791458 832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:49.791486 832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:07:49.791518 832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:07:49.793040 832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:36401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:36401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:07:49.793344 832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:49.793606 832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:49.799329 840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:49.799401 837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:49.799364 838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:49.799481 832 server_base.cc:1034] running on GCE node
I20260620 19:07:49.799706 832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:49.800009 832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:49.801177 832 hybrid_clock.cc:648] HybridClock initialized: now 1781982469801161 us; error 27 us; skew 500 ppm
Jun 20 19:07:49 dist-test-slave-wd30 krb5kdc[819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:49.803714 832 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:07:49.804212 832 webserver.cc:460] Webserver started at http://127.0.203.62:42895/ using document root <none> and password file <none>
I20260620 19:07:49.804450 832 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:49.804518 832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:49.804672 832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:49.805770 832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "252086e62738427fa7fd771e02fca12c"
format_stamp: "Formatted at 2026-06-20 19:07:49 on dist-test-slave-wd30"
server_key: "ad19b06c07bdfbc434def26dce27e58a"
server_key_iv: "76a29f641c91161c45dbb9833cb687bf"
server_key_version: "encryptionkey@0"
I20260620 19:07:49.806157 832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "252086e62738427fa7fd771e02fca12c"
format_stamp: "Formatted at 2026-06-20 19:07:49 on dist-test-slave-wd30"
server_key: "ad19b06c07bdfbc434def26dce27e58a"
server_key_iv: "76a29f641c91161c45dbb9833cb687bf"
server_key_version: "encryptionkey@0"
I20260620 19:07:49.807612 832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:07:49.808426 847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:49.808619 832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:07:49.808741 832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "252086e62738427fa7fd771e02fca12c"
format_stamp: "Formatted at 2026-06-20 19:07:49 on dist-test-slave-wd30"
server_key: "ad19b06c07bdfbc434def26dce27e58a"
server_key_iv: "76a29f641c91161c45dbb9833cb687bf"
server_key_version: "encryptionkey@0"
I20260620 19:07:49.808817 832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:49.816483 832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:49.818307 832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:49.818500 832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:49.822682 832 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:36401
I20260620 19:07:49.822726 899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:36401 every 8 connection(s)
I20260620 19:07:49.823108 832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:07:49.824119 900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:49.826495 900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c: Bootstrap starting.
I20260620 19:07:49.827124 900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:49.827462 900 log.cc:824] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:49.828418 900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c: No bootstrap required, opened a new log
I20260620 19:07:49.830029 900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252086e62738427fa7fd771e02fca12c" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36401 } }
I20260620 19:07:49.830193 900 raft_consensus.cc:374] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:49.830240 900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252086e62738427fa7fd771e02fca12c, State: Initialized, Role: FOLLOWER
I20260620 19:07:49.830392 900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252086e62738427fa7fd771e02fca12c" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36401 } }
I20260620 19:07:49.830476 900 raft_consensus.cc:388] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:07:49.830520 900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:07:49.830569 900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:49.831092 900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252086e62738427fa7fd771e02fca12c" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36401 } }
I20260620 19:07:49.831243 900 leader_election.cc:302] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 252086e62738427fa7fd771e02fca12c; no voters:
I20260620 19:07:49.831419 900 leader_election.cc:288] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:07:49.831455 903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:07:49.831568 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 832
I20260620 19:07:49.831676 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:07:49.831668 903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c [term 1 LEADER]: Becoming Leader. State: Replica: 252086e62738427fa7fd771e02fca12c, State: Running, Role: LEADER
I20260620 19:07:49.831686 900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:07:49.831751 903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252086e62738427fa7fd771e02fca12c" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36401 } }
I20260620 19:07:49.832093 903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 252086e62738427fa7fd771e02fca12c. Latest consensus state: current_term: 1 leader_uuid: "252086e62738427fa7fd771e02fca12c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252086e62738427fa7fd771e02fca12c" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36401 } } }
I20260620 19:07:49.832147 903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c [sys.catalog]: This master's current role is: LEADER
I20260620 19:07:49.832350 904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "252086e62738427fa7fd771e02fca12c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252086e62738427fa7fd771e02fca12c" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36401 } } }
I20260620 19:07:49.832448 904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c [sys.catalog]: This master's current role is: LEADER
I20260620 19:07:49.832479 910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:07:49.832499 812 external_mini_cluster.cc:1428] Setting key 87339a462d97d1ee1ef4d847e40dcfa0
I20260620 19:07:49.833007 910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:07:49.834784 910 catalog_manager.cc:1347] Generated new cluster ID: e4f78233cb0f4b66b9f66af4db279aff
I20260620 19:07:49.834838 910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:07:49.841116 910 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:07:49.841729 910 catalog_manager.cc:1504] Loading token signing keys...
Jun 20 19:07:49 dist-test-slave-wd30 krb5kdc[819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:07:49.847877 910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 252086e62738427fa7fd771e02fca12c: Generated new TSK 0
I20260620 19:07:49.848083 910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:07:49.850839 922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:49.833556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45844 (local address 127.0.203.62:36401)
0620 19:07:49.833793 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:49.833802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:49.834101 (+ 299us) server_negotiation.cc:407] Connection header received
0620 19:07:49.836324 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:49.836333 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:49.836404 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:49.836556 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:49.838470 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:49.840026 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:49.841534 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:49.842167 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:49.842282 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:49.846604 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:49.846608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:49.846608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:49.846621 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:49.848838 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:49.849264 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:49.849267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:49.849267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:49.849390 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:49.849646 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:49.849646 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:49.849647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:49.849947 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:49.850097 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:49.850444 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 19:07:49.850616 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
I20260620 19:07:49.878273 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36401
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:49.960422 926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:49.960659 926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:49.960701 926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:49.961859 926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:49.961928 926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:49.961972 926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:07:49.963567 926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36401
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:07:49.963850 926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:49.964141 926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:49.969372 934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:49.969388 931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:49.969393 932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:49.969406 926 server_base.cc:1034] running on GCE node
I20260620 19:07:49.969748 926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:49.970018 926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:49.971194 926 hybrid_clock.cc:648] HybridClock initialized: now 1781982469971175 us; error 39 us; skew 500 ppm
Jun 20 19:07:49 dist-test-slave-wd30 krb5kdc[819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:49.973618 926 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:07:49.974049 926 webserver.cc:460] Webserver started at http://127.0.203.1:46619/ using document root <none> and password file <none>
I20260620 19:07:49.974283 926 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:49.974351 926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:49.974504 926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:49.975585 926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "1f76728f50164705afee6f58dfc62b72"
format_stamp: "Formatted at 2026-06-20 19:07:49 on dist-test-slave-wd30"
server_key: "f2a911867bfe83a4c91f212a460043e0"
server_key_iv: "2425bc454e1786f0843e564ce3f3c5c4"
server_key_version: "encryptionkey@0"
I20260620 19:07:49.975934 926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "1f76728f50164705afee6f58dfc62b72"
format_stamp: "Formatted at 2026-06-20 19:07:49 on dist-test-slave-wd30"
server_key: "f2a911867bfe83a4c91f212a460043e0"
server_key_iv: "2425bc454e1786f0843e564ce3f3c5c4"
server_key_version: "encryptionkey@0"
I20260620 19:07:49.977233 926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:07:49.977946 941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:49.978142 926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:07:49.978242 926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "1f76728f50164705afee6f58dfc62b72"
format_stamp: "Formatted at 2026-06-20 19:07:49 on dist-test-slave-wd30"
server_key: "f2a911867bfe83a4c91f212a460043e0"
server_key_iv: "2425bc454e1786f0843e564ce3f3c5c4"
server_key_version: "encryptionkey@0"
I20260620 19:07:49.978319 926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:49.991381 926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:49.993700 926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:49.993860 926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:49.994091 926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:49.994395 926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:49.994450 926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:49.994494 926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:49.994524 926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:50.000299 926 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:46263
I20260620 19:07:50.000342 1054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:46263 every 8 connection(s)
I20260620 19:07:50.000658 926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:07:50.003517 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 926
I20260620 19:07:50.003588 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:07:50.003742 812 external_mini_cluster.cc:1428] Setting key d8833bac51d4a98ee3350b006c2a69ca
Jun 20 19:07:50 dist-test-slave-wd30 krb5kdc[819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:07:50.010535 922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:50.001355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:59169 (local address 127.0.203.62:36401)
0620 19:07:50.001423 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:50.001425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:50.001610 (+ 185us) server_negotiation.cc:407] Connection header received
0620 19:07:50.002100 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:50.002101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:50.002112 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:50.002180 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:50.003045 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.003673 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.004923 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.005134 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.005227 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:50.007658 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:50.007660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:50.007661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:50.007667 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:50.009118 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:50.009530 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:50.009532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:50.009532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:50.009593 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:50.009857 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:50.009858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:50.009858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:50.009989 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:50.010065 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:50.010337 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 19:07:50.010441 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:07:50.010703 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:50.001327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36401 (local address 127.0.203.1:59169)
0620 19:07:50.001472 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:50.001515 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:50.002013 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:50.002233 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:50.002236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:50.002772 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:50.002958 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:50.002963 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.003767 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.003769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:50.004824 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:50.004827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.005250 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.005250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:50.005370 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:50.005915 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:50.005924 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:50.007557 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:50.009201 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:50.009207 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:50.009210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:50.009433 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:50.009669 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:50.009670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:50.009670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:50.009786 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:50.010056 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:50.010062 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:50.010202 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 19:07:50.010305 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 19:07:50.010460 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260620 19:07:50.011183 1055 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36401
I20260620 19:07:50.011271 1055 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:50.011462 1055 heartbeater.cc:511] Master 127.0.203.62:36401 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.203.2@KRBTEST.COM; defaulting to no policy
I20260620 19:07:50.012106 864 ts_manager.cc:194] Registered new tserver with Master: 1f76728f50164705afee6f58dfc62b72 (127.0.203.1:46263)
I20260620 19:07:50.012861 864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:59169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
I20260620 19:07:50.027628 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36401
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:50.109256 1060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:50.109462 1060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:50.109484 1060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:50.110538 1060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:50.110571 1060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:50.110597 1060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:07:50.112206 1060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36401
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:07:50.112429 1060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:50.112617 1060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:50.117586 1068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:50.117626 1065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:50.117666 1060 server_base.cc:1034] running on GCE node
W20260620 19:07:50.117607 1066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:50.117997 1060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:50.118275 1060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:50.119429 1060 hybrid_clock.cc:648] HybridClock initialized: now 1781982470119409 us; error 36 us; skew 500 ppm
Jun 20 19:07:50 dist-test-slave-wd30 krb5kdc[819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:50.122013 1060 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:07:50.122475 1060 webserver.cc:460] Webserver started at http://127.0.203.2:46303/ using document root <none> and password file <none>
I20260620 19:07:50.122696 1060 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:50.122740 1060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:50.122906 1060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:50.124003 1060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "72a680134b2640b7b0d3db57dcb7c810"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-wd30"
server_key: "ed13145ece2801aafcc8b75602a2a19d"
server_key_iv: "e703f9e6117b0eb7b09515ff0d17aa90"
server_key_version: "encryptionkey@0"
I20260620 19:07:50.124370 1060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "72a680134b2640b7b0d3db57dcb7c810"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-wd30"
server_key: "ed13145ece2801aafcc8b75602a2a19d"
server_key_iv: "e703f9e6117b0eb7b09515ff0d17aa90"
server_key_version: "encryptionkey@0"
I20260620 19:07:50.125700 1060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:07:50.126468 1075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:50.126674 1060 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:07:50.126775 1060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "72a680134b2640b7b0d3db57dcb7c810"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-wd30"
server_key: "ed13145ece2801aafcc8b75602a2a19d"
server_key_iv: "e703f9e6117b0eb7b09515ff0d17aa90"
server_key_version: "encryptionkey@0"
I20260620 19:07:50.126854 1060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:50.137140 1060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:50.139478 1060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:50.139639 1060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:50.139864 1060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:50.140177 1060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:50.140209 1060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:50.140231 1060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:50.140278 1060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:50.147070 1060 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:39321
I20260620 19:07:50.147138 1188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:39321 every 8 connection(s)
I20260620 19:07:50.147439 1060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:07:50.151947 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 1060
I20260620 19:07:50.152057 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:07:50.152244 812 external_mini_cluster.cc:1428] Setting key c7393e74e4022b80d6e29d7c28888bb7
Jun 20 19:07:50 dist-test-slave-wd30 krb5kdc[819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:07:50.156931 922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:50.148145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:56519 (local address 127.0.203.62:36401)
0620 19:07:50.148230 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:50.148232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:50.148432 (+ 200us) server_negotiation.cc:407] Connection header received
0620 19:07:50.148904 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:50.148905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:50.148914 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:50.149006 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:50.149911 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.150420 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.151473 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.151656 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.151709 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:50.153946 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:50.153947 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:50.153948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:50.153965 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:50.155436 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:50.155927 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:50.155927 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:50.155928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:50.155994 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:50.156295 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:50.156296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:50.156296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:50.156407 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:50.156488 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:50.156761 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 19:07:50.156842 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 19:07:50.157116 1191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:50.148151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36401 (local address 127.0.203.2:56519)
0620 19:07:50.148305 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:50.148347 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:50.148819 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:50.149071 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:50.149074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:50.149638 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:50.149834 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:50.149838 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.150500 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.150500 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:50.151404 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:50.151406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.151727 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.151728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:50.151837 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:50.152425 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:50.152437 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:50.153859 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:50.155519 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:50.155525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:50.155528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:50.155847 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:50.156063 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:50.156065 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:50.156065 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:50.156229 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:50.156481 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:50.156487 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:50.156627 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 19:07:50.156731 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 19:07:50.156889 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260620 19:07:50.157773 1189 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36401
I20260620 19:07:50.157881 1189 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:50.158083 1189 heartbeater.cc:511] Master 127.0.203.62:36401 requested a full tablet report, sending...
I20260620 19:07:50.158541 864 ts_manager.cc:194] Registered new tserver with Master: 72a680134b2640b7b0d3db57dcb7c810 (127.0.203.2:39321)
I20260620 19:07:50.158980 864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:56519
WARNING: no policy specified for kudu/127.0.203.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
I20260620 19:07:50.176167 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36401
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:50.260517 1194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:50.260715 1194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:50.260736 1194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:50.261827 1194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:50.261862 1194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:50.261888 1194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:07:50.263296 1194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36401
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:07:50.263496 1194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:50.263665 1194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:50.268420 1202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:50.268427 1199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:50.268427 1200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:50.268455 1194 server_base.cc:1034] running on GCE node
I20260620 19:07:50.268893 1194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:50.269150 1194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:50.270315 1194 hybrid_clock.cc:648] HybridClock initialized: now 1781982470270308 us; error 34 us; skew 500 ppm
Jun 20 19:07:50 dist-test-slave-wd30 krb5kdc[819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:50.272784 1194 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:07:50.273267 1194 webserver.cc:460] Webserver started at http://127.0.203.3:40811/ using document root <none> and password file <none>
I20260620 19:07:50.273492 1194 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:50.273561 1194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:50.273707 1194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:50.274695 1194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "c0c0bbebc6a5494084aae1ca17ade096"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-wd30"
server_key: "ef7e01be7ee05bf4e6add9522a8f2f17"
server_key_iv: "c4f1644c97f11cd1a17b261abf4b3c1d"
server_key_version: "encryptionkey@0"
I20260620 19:07:50.275053 1194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "c0c0bbebc6a5494084aae1ca17ade096"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-wd30"
server_key: "ef7e01be7ee05bf4e6add9522a8f2f17"
server_key_iv: "c4f1644c97f11cd1a17b261abf4b3c1d"
server_key_version: "encryptionkey@0"
I20260620 19:07:50.276365 1194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:07:50.277170 1209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:50.277469 1194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:07:50.277590 1194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "c0c0bbebc6a5494084aae1ca17ade096"
format_stamp: "Formatted at 2026-06-20 19:07:50 on dist-test-slave-wd30"
server_key: "ef7e01be7ee05bf4e6add9522a8f2f17"
server_key_iv: "c4f1644c97f11cd1a17b261abf4b3c1d"
server_key_version: "encryptionkey@0"
I20260620 19:07:50.277668 1194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:50.299468 1194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:50.301473 1194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:50.301651 1194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:50.301913 1194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:50.302237 1194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:50.302294 1194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:50.302330 1194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:50.302377 1194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:50.308323 1194 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:40155
I20260620 19:07:50.308413 1322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:40155 every 8 connection(s)
I20260620 19:07:50.308729 1194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:07:50.311041 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 1194
I20260620 19:07:50.311167 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:07:50.311354 812 external_mini_cluster.cc:1428] Setting key c5542b9454ca71decc87f37800a5053d
Jun 20 19:07:50 dist-test-slave-wd30 krb5kdc[819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:07:50.317837 922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:50.309388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:49355 (local address 127.0.203.62:36401)
0620 19:07:50.309450 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:50.309452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:50.309674 (+ 222us) server_negotiation.cc:407] Connection header received
0620 19:07:50.310149 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:50.310150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:50.310160 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:50.310210 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:50.311098 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.311832 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.313112 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.313296 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.313351 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:50.315430 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:50.315431 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:50.315431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:50.315437 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:50.316700 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:50.317092 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:50.317092 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:50.317092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:50.317139 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:50.317349 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:50.317350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:50.317350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:50.317442 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:50.317491 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:50.317719 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 19:07:50.317774 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:07:50.318089 1325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:50.309391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36401 (local address 127.0.203.3:49355)
0620 19:07:50.309553 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:50.309588 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:50.310068 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:50.310263 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:50.310266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:50.310811 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:50.310993 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:50.310997 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.311929 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.311930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:50.313043 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:50.313046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.313366 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.313367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:50.313463 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:50.313984 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:50.313992 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:50.315353 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:50.316777 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:50.316783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:50.316784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:50.317025 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:50.317198 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:50.317199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:50.317199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:50.317296 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:50.317497 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:50.317502 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:50.317610 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 19:07:50.317692 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 19:07:50.317805 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260620 19:07:50.318536 1323 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36401
I20260620 19:07:50.318612 1323 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:50.318749 1323 heartbeater.cc:511] Master 127.0.203.62:36401 requested a full tablet report, sending...
I20260620 19:07:50.319077 864 ts_manager.cc:194] Registered new tserver with Master: c0c0bbebc6a5494084aae1ca17ade096 (127.0.203.3:40155)
I20260620 19:07:50.319458 864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:49355
I20260620 19:07:50.323405 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:07:50 dist-test-slave-wd30 krb5kdc[819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982470, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:07:50 dist-test-slave-wd30 krb5kdc[819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982470, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.203.1@KRBTEST.COM
I20260620 19:07:50.346190 1330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:50.338693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51414 (local address 127.0.203.1:46263)
0620 19:07:50.338868 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:50.338874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:50.338885 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:07:50.338911 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:50.338913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:50.338928 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:50.339020 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:50.339613 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.340326 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.341044 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.341372 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.341436 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:50.343692 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:50.343700 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:50.343701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:50.343707 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:50.345093 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:50.345399 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:50.345402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:50.345402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:50.345465 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:50.345715 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:50.345715 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:50.345715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:50.345836 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:50.345943 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:50.346013 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 19:07:50.346072 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
W20260620 19:07:50.346583 969 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51414
Jun 20 19:07:50 dist-test-slave-wd30 krb5kdc[819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:07:50 dist-test-slave-wd30 krb5kdc[819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.1@KRBTEST.COM
I20260620 19:07:50.368865 1330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:50.361866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.0.203.1:46263)
0620 19:07:50.361928 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:50.361929 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:50.361971 (+ 42us) server_negotiation.cc:407] Connection header received
0620 19:07:50.362040 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:50.362041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:50.362052 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:50.362120 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:50.362670 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.363234 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.363920 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:50.364112 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:50.364161 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:50.366299 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:50.366301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:50.366302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:50.366310 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:50.367725 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:50.368095 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:50.368096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:50.368097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:50.368156 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:50.368353 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:50.368354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:50.368354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:50.368453 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:50.368591 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:50.368672 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 19:07:50.368756 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:07:50.369401 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 926
I20260620 19:07:50.374365 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 1060
I20260620 19:07:50.379354 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 1194
I20260620 19:07:50.383800 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 832
2026-06-20T19:07:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1853 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:07:50 dist-test-slave-wd30 krb5kdc[1337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:07:50 dist-test-slave-wd30 krb5kdc[1337](info): set up 2 sockets
Jun 20 19:07:50 dist-test-slave-wd30 krb5kdc[1337](info): commencing operation
krb5kdc: starting...
W20260620 19:07:51.418918 812 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:07:51 dist-test-slave-wd30 krb5kdc[1337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:07:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:07:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:07:51.543581 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:40793
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:40793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:51.629773 1350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:51.630062 1350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:51.630102 1350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:51.631340 1350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:07:51.631421 1350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:51.631448 1350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:51.631469 1350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:07:51.631500 1350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:07:51.633064 1350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:40793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:40793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:07:51.633379 1350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:51.633653 1350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:07:51.638849 1350 server_base.cc:1034] running on GCE node
W20260620 19:07:51.638811 1355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:51.638818 1358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:51.638837 1356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:51.639307 1350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:51.639618 1350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:51.640800 1350 hybrid_clock.cc:648] HybridClock initialized: now 1781982471640782 us; error 39 us; skew 500 ppm
Jun 20 19:07:51 dist-test-slave-wd30 krb5kdc[1337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:51.646025 1350 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:07:51.646545 1350 webserver.cc:460] Webserver started at http://127.0.203.62:36139/ using document root <none> and password file <none>
I20260620 19:07:51.646800 1350 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:51.646873 1350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:51.647029 1350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:51.648190 1350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "c062b18b96bc4d0f974f0709d22eb226"
format_stamp: "Formatted at 2026-06-20 19:07:51 on dist-test-slave-wd30"
server_key: "a81f99903ce13b6bb21758b9189fecd7"
server_key_iv: "802e6b36d1745136e8db29e8ccae3ab8"
server_key_version: "encryptionkey@0"
I20260620 19:07:51.648630 1350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "c062b18b96bc4d0f974f0709d22eb226"
format_stamp: "Formatted at 2026-06-20 19:07:51 on dist-test-slave-wd30"
server_key: "a81f99903ce13b6bb21758b9189fecd7"
server_key_iv: "802e6b36d1745136e8db29e8ccae3ab8"
server_key_version: "encryptionkey@0"
I20260620 19:07:51.650066 1350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:07:51.650846 1365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:51.651088 1350 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:07:51.651227 1350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "c062b18b96bc4d0f974f0709d22eb226"
format_stamp: "Formatted at 2026-06-20 19:07:51 on dist-test-slave-wd30"
server_key: "a81f99903ce13b6bb21758b9189fecd7"
server_key_iv: "802e6b36d1745136e8db29e8ccae3ab8"
server_key_version: "encryptionkey@0"
I20260620 19:07:51.651329 1350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:51.669342 1350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:51.671828 1350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:51.672005 1350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:51.676261 1350 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:40793
I20260620 19:07:51.676299 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:40793 every 8 connection(s)
I20260620 19:07:51.676704 1350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:07:51.677642 1418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:51.679095 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 1350
I20260620 19:07:51.679255 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:07:51.679418 812 external_mini_cluster.cc:1428] Setting key 8235b3ba16cb1141983d729332b5c6fd
I20260620 19:07:51.680236 1418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226: Bootstrap starting.
I20260620 19:07:51.680833 1418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:51.681165 1418 log.cc:824] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:51.681856 1418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226: No bootstrap required, opened a new log
I20260620 19:07:51.683734 1418 raft_consensus.cc:348] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c062b18b96bc4d0f974f0709d22eb226" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 40793 } }
I20260620 19:07:51.683900 1418 raft_consensus.cc:374] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:51.683943 1418 raft_consensus.cc:729] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c062b18b96bc4d0f974f0709d22eb226, State: Initialized, Role: FOLLOWER
I20260620 19:07:51.684068 1418 consensus_queue.cc:260] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c062b18b96bc4d0f974f0709d22eb226" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 40793 } }
I20260620 19:07:51.684176 1418 raft_consensus.cc:388] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:07:51.684229 1418 raft_consensus.cc:482] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:07:51.684269 1418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:51.684937 1418 raft_consensus.cc:504] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c062b18b96bc4d0f974f0709d22eb226" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 40793 } }
I20260620 19:07:51.685065 1418 leader_election.cc:302] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c062b18b96bc4d0f974f0709d22eb226; no voters:
I20260620 19:07:51.685281 1418 leader_election.cc:288] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:07:51.685343 1423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:07:51.685508 1423 raft_consensus.cc:686] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226 [term 1 LEADER]: Becoming Leader. State: Replica: c062b18b96bc4d0f974f0709d22eb226, State: Running, Role: LEADER
I20260620 19:07:51.685580 1418 sys_catalog.cc:564] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:07:51.685736 1423 consensus_queue.cc:237] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c062b18b96bc4d0f974f0709d22eb226" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 40793 } }
Jun 20 19:07:51 dist-test-slave-wd30 krb5kdc[1337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:07:51.686137 1423 sys_catalog.cc:455] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c062b18b96bc4d0f974f0709d22eb226. Latest consensus state: current_term: 1 leader_uuid: "c062b18b96bc4d0f974f0709d22eb226" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c062b18b96bc4d0f974f0709d22eb226" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 40793 } } }
I20260620 19:07:51.686231 1423 sys_catalog.cc:458] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226 [sys.catalog]: This master's current role is: LEADER
I20260620 19:07:51.686148 1424 sys_catalog.cc:455] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c062b18b96bc4d0f974f0709d22eb226" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c062b18b96bc4d0f974f0709d22eb226" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 40793 } } }
I20260620 19:07:51.686302 1424 sys_catalog.cc:458] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226 [sys.catalog]: This master's current role is: LEADER
I20260620 19:07:51.687182 1429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:07:51.687747 1429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:07:51.689693 1429 catalog_manager.cc:1347] Generated new cluster ID: 1b4e3c2a54d64bd38c4a9bfd40d7dcd7
I20260620 19:07:51.689744 1429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:07:51.690850 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.680005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41924 (local address 127.0.203.62:40793)
0620 19:07:51.680170 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:51.680179 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:51.680280 (+ 101us) server_negotiation.cc:407] Connection header received
0620 19:07:51.680774 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:51.680781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:51.680830 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:51.680982 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:51.681614 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.682541 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.683384 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.683806 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.683931 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.686757 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:51.686762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:51.686762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:51.686776 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:51.688837 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:51.689298 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:51.689301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:51.689301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:51.689411 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:51.689624 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:51.689626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:51.689626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:51.689942 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:51.690080 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:51.690503 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 19:07:51.690633 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260620 19:07:51.712548 1429 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:07:51.713132 1429 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:07:51.721444 1429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c062b18b96bc4d0f974f0709d22eb226: Generated new TSK 0
I20260620 19:07:51.721668 1429 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
I20260620 19:07:51.762053 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:40793
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:51.852991 1444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:51.853225 1444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:51.853250 1444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:51.854537 1444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:51.854576 1444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:51.854604 1444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:07:51.856160 1444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:40793
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:07:51.856410 1444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:51.856666 1444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:51.861944 1449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:51.862067 1450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:51.861946 1452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:51.862190 1444 server_base.cc:1034] running on GCE node
I20260620 19:07:51.862412 1444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:51.862699 1444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:51.863855 1444 hybrid_clock.cc:648] HybridClock initialized: now 1781982471863841 us; error 31 us; skew 500 ppm
Jun 20 19:07:51 dist-test-slave-wd30 krb5kdc[1337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:51.866410 1444 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:07:51.866798 1444 webserver.cc:460] Webserver started at http://127.0.203.1:37705/ using document root <none> and password file <none>
I20260620 19:07:51.866983 1444 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:51.867026 1444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:51.867165 1444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:51.868207 1444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "9e5d2a44600240ba8901aaf4051650b2"
format_stamp: "Formatted at 2026-06-20 19:07:51 on dist-test-slave-wd30"
server_key: "f5a1dc7adf3517ad98a3e5f6065ca8aa"
server_key_iv: "0cfeba079428ff0b4490782f874baa2c"
server_key_version: "encryptionkey@0"
I20260620 19:07:51.868511 1444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "9e5d2a44600240ba8901aaf4051650b2"
format_stamp: "Formatted at 2026-06-20 19:07:51 on dist-test-slave-wd30"
server_key: "f5a1dc7adf3517ad98a3e5f6065ca8aa"
server_key_iv: "0cfeba079428ff0b4490782f874baa2c"
server_key_version: "encryptionkey@0"
I20260620 19:07:51.869823 1444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:07:51.870518 1459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:51.870710 1444 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:07:51.870829 1444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "9e5d2a44600240ba8901aaf4051650b2"
format_stamp: "Formatted at 2026-06-20 19:07:51 on dist-test-slave-wd30"
server_key: "f5a1dc7adf3517ad98a3e5f6065ca8aa"
server_key_iv: "0cfeba079428ff0b4490782f874baa2c"
server_key_version: "encryptionkey@0"
I20260620 19:07:51.870908 1444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:51.905546 1444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:51.907599 1444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:51.907774 1444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:51.908018 1444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:51.908399 1444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:51.908454 1444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:51.908495 1444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:51.908522 1444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:51.914188 1444 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:34921
I20260620 19:07:51.914234 1572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:34921 every 8 connection(s)
I20260620 19:07:51.914551 1444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:07:51.918588 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 1444
I20260620 19:07:51.918684 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:07:51.918840 812 external_mini_cluster.cc:1428] Setting key df8bf650f51f3d87b289cfdc2c768280
Jun 20 19:07:51 dist-test-slave-wd30 krb5kdc[1337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:07:51.925067 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.915343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:38451 (local address 127.0.203.62:40793)
0620 19:07:51.915417 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:51.915419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:51.915617 (+ 198us) server_negotiation.cc:407] Connection header received
0620 19:07:51.916141 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:51.916142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:51.916153 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:51.916211 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:51.917168 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.917692 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.918935 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.919264 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.919303 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.921851 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:51.921853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:51.921853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:51.921860 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:51.923632 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:51.924075 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:51.924077 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:51.924077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:51.924140 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:51.924409 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:51.924410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:51.924410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:51.924530 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:51.924608 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:51.924887 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 19:07:51.924977 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:07:51.925309 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:51.915340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:40793 (local address 127.0.203.1:38451)
0620 19:07:51.915495 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:51.915533 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:51.916039 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:51.916291 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:51.916295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:51.916864 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:51.917067 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:51.917072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.917804 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.917805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:51.918824 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:51.918827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:51.919401 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:51.919402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:51.919564 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:51.920106 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:51.920117 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:51.921749 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:51.923739 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:51.923746 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:51.923749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:51.923980 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:51.924227 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:51.924228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:51.924229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:51.924340 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:51.924598 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:51.924605 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:51.924752 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 19:07:51.924866 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 19:07:51.925028 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260620 19:07:51.926033 1573 heartbeater.cc:347] Connected to a master server at 127.0.203.62:40793
I20260620 19:07:51.926144 1573 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:51.926329 1573 heartbeater.cc:511] Master 127.0.203.62:40793 requested a full tablet report, sending...
I20260620 19:07:51.926918 1382 ts_manager.cc:194] Registered new tserver with Master: 9e5d2a44600240ba8901aaf4051650b2 (127.0.203.1:34921)
WARNING: no policy specified for kudu/127.0.203.2@KRBTEST.COM; defaulting to no policy
I20260620 19:07:51.927729 1382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:38451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
I20260620 19:07:51.945556 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:40793
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:52.025555 1578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:52.025831 1578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:52.025890 1578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:52.027072 1578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:52.027170 1578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:52.027225 1578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:07:52.028671 1578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:40793
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:07:52.028946 1578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:52.029175 1578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:52.034098 1583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:52.034137 1584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:52.034252 1578 server_base.cc:1034] running on GCE node
W20260620 19:07:52.034106 1586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:52.034634 1578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:52.034900 1578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:52.036062 1578 hybrid_clock.cc:648] HybridClock initialized: now 1781982472036058 us; error 35 us; skew 500 ppm
Jun 20 19:07:52 dist-test-slave-wd30 krb5kdc[1337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:52.038935 1578 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:07:52.039458 1578 webserver.cc:460] Webserver started at http://127.0.203.2:44727/ using document root <none> and password file <none>
I20260620 19:07:52.039641 1578 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:52.039681 1578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:52.039819 1578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:52.040858 1578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "993df3a8d2154f7da1c7eecf236923bf"
format_stamp: "Formatted at 2026-06-20 19:07:52 on dist-test-slave-wd30"
server_key: "72c1c18ba6aadff7ae1c7009f167758b"
server_key_iv: "c514a5f337e401ae2d6c4f9ed64869af"
server_key_version: "encryptionkey@0"
I20260620 19:07:52.041157 1578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "993df3a8d2154f7da1c7eecf236923bf"
format_stamp: "Formatted at 2026-06-20 19:07:52 on dist-test-slave-wd30"
server_key: "72c1c18ba6aadff7ae1c7009f167758b"
server_key_iv: "c514a5f337e401ae2d6c4f9ed64869af"
server_key_version: "encryptionkey@0"
I20260620 19:07:52.042419 1578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:07:52.043097 1593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:52.043339 1578 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:07:52.043428 1578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "993df3a8d2154f7da1c7eecf236923bf"
format_stamp: "Formatted at 2026-06-20 19:07:52 on dist-test-slave-wd30"
server_key: "72c1c18ba6aadff7ae1c7009f167758b"
server_key_iv: "c514a5f337e401ae2d6c4f9ed64869af"
server_key_version: "encryptionkey@0"
I20260620 19:07:52.043485 1578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:52.059465 1578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:52.061508 1578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:52.061622 1578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:52.061811 1578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:52.062140 1578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:52.062176 1578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:52.062199 1578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:52.062255 1578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:52.068005 1578 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:43453
I20260620 19:07:52.068063 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:43453 every 8 connection(s)
I20260620 19:07:52.068387 1578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:07:52.070320 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 1578
I20260620 19:07:52.070412 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:07:52.070591 812 external_mini_cluster.cc:1428] Setting key 58ebeba18c80f5dd84365a23db4d5fa1
Jun 20 19:07:52 dist-test-slave-wd30 krb5kdc[1337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.203.3@KRBTEST.COM; defaulting to no policy
I20260620 19:07:52.079815 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.069320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:53145 (local address 127.0.203.62:40793)
0620 19:07:52.069380 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:52.069382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:52.069602 (+ 220us) server_negotiation.cc:407] Connection header received
0620 19:07:52.070094 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:52.070095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:52.070105 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:52.070206 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:52.071457 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.072241 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.073777 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.073975 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.074029 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.076816 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:52.076818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:52.076819 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:52.076828 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:52.078300 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:52.078764 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:52.078765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:52.078766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:52.078827 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:52.079102 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:52.079102 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:52.079103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:52.079261 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:52.079336 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:52.079623 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 19:07:52.079716 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:07:52.080020 1709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.069256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:40793 (local address 127.0.203.2:53145)
0620 19:07:52.069453 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:52.069497 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:52.070009 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:52.070283 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:52.070286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:52.071053 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:52.071382 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.071388 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.072334 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.072335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.073675 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.073678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.074066 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.074067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.074238 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.074918 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:52.074930 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:52.076703 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:52.078422 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:52.078429 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:52.078431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:52.078666 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:52.078895 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:52.078896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:52.078896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:52.079024 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:52.079341 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:52.079348 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:52.079492 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 19:07:52.079603 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 19:07:52.079773 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":117,"threads_started":1}
I20260620 19:07:52.080547 1707 heartbeater.cc:347] Connected to a master server at 127.0.203.62:40793
I20260620 19:07:52.080650 1707 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:52.080850 1707 heartbeater.cc:511] Master 127.0.203.62:40793 requested a full tablet report, sending...
I20260620 19:07:52.081241 1382 ts_manager.cc:194] Registered new tserver with Master: 993df3a8d2154f7da1c7eecf236923bf (127.0.203.2:43453)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.3@KRBTEST.COM" created.
I20260620 19:07:52.081784 1382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:53145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
I20260620 19:07:52.095875 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:40793
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:52.177176 1712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:52.177403 1712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:52.177428 1712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:52.178632 1712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:52.178678 1712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:52.178706 1712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:07:52.180167 1712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:40793
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:07:52.180418 1712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:52.180667 1712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:52.185665 1717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:52.185662 1718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:52.185855 1712 server_base.cc:1034] running on GCE node
W20260620 19:07:52.185657 1720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:52.186157 1712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:52.186496 1712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:52.187721 1712 hybrid_clock.cc:648] HybridClock initialized: now 1781982472187705 us; error 26 us; skew 500 ppm
Jun 20 19:07:52 dist-test-slave-wd30 krb5kdc[1337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:52.190225 1712 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:07:52.190749 1712 webserver.cc:460] Webserver started at http://127.0.203.3:42813/ using document root <none> and password file <none>
I20260620 19:07:52.190974 1712 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:52.191042 1712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:52.191227 1712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:52.192291 1712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "247c68dd5fe44eec9273a46750e5a453"
format_stamp: "Formatted at 2026-06-20 19:07:52 on dist-test-slave-wd30"
server_key: "e81ae2b2c9c22a61a4120cffa6f79041"
server_key_iv: "c5ed0b541a1319d1f5056f78423dd9d8"
server_key_version: "encryptionkey@0"
I20260620 19:07:52.192665 1712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "247c68dd5fe44eec9273a46750e5a453"
format_stamp: "Formatted at 2026-06-20 19:07:52 on dist-test-slave-wd30"
server_key: "e81ae2b2c9c22a61a4120cffa6f79041"
server_key_iv: "c5ed0b541a1319d1f5056f78423dd9d8"
server_key_version: "encryptionkey@0"
I20260620 19:07:52.193969 1712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:07:52.194707 1727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:52.194886 1712 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:07:52.194983 1712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "247c68dd5fe44eec9273a46750e5a453"
format_stamp: "Formatted at 2026-06-20 19:07:52 on dist-test-slave-wd30"
server_key: "e81ae2b2c9c22a61a4120cffa6f79041"
server_key_iv: "c5ed0b541a1319d1f5056f78423dd9d8"
server_key_version: "encryptionkey@0"
I20260620 19:07:52.195070 1712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:52.222887 1712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:52.225937 1712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:52.226121 1712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:52.226449 1712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:07:52.226930 1712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:52.226980 1712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:52.227033 1712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:52.227065 1712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:52.233453 1712 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:42777
I20260620 19:07:52.233556 1840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:42777 every 8 connection(s)
I20260620 19:07:52.233873 1712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:07:52 dist-test-slave-wd30 krb5kdc[1337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:07:52.240808 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 1712
I20260620 19:07:52.240911 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:07:52.241091 812 external_mini_cluster.cc:1428] Setting key c230c898e3e8004b8e3826d58cddba6b
I20260620 19:07:52.243606 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.234674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:42137 (local address 127.0.203.62:40793)
0620 19:07:52.234736 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:52.234738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:52.234904 (+ 166us) server_negotiation.cc:407] Connection header received
0620 19:07:52.235478 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:52.235479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:52.235488 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:52.235552 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:52.236481 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.236983 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.237946 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.238120 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.238169 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.240427 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:52.240429 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:52.240429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:52.240436 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:52.242156 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:52.242559 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:52.242560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:52.242561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:52.242625 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:52.242882 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:52.242883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:52.242883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:52.243006 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:52.243078 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:52.243434 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 19:07:52.243513 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:07:52.243816 1843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.234616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:40793 (local address 127.0.203.3:42137)
0620 19:07:52.234778 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:52.234821 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:52.235370 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:52.235642 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:52.235645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:52.236215 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:52.236406 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.236410 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.237061 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.237062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.237875 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.237877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.238223 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.238224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.238334 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.238887 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:52.238896 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:52.240341 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:52.242244 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:52.242251 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:52.242254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:52.242480 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:52.242688 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:52.242689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:52.242690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:52.242807 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:52.243069 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:52.243075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:52.243291 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 19:07:52.243411 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 19:07:52.243572 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260620 19:07:52.244259 1841 heartbeater.cc:347] Connected to a master server at 127.0.203.62:40793
I20260620 19:07:52.244352 1841 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:52.244498 1841 heartbeater.cc:511] Master 127.0.203.62:40793 requested a full tablet report, sending...
I20260620 19:07:52.244814 1382 ts_manager.cc:194] Registered new tserver with Master: 247c68dd5fe44eec9273a46750e5a453 (127.0.203.3:42777)
I20260620 19:07:52.245141 1382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:42137
I20260620 19:07:52.252884 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:07:52.260852 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.254746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41940 (local address 127.0.203.62:40793)
0620 19:07:52.254822 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:52.254824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:52.254831 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:07:52.254850 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:52.254851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:52.254873 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:52.254964 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:52.255641 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.256124 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.256839 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.257016 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.257066 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.258295 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:52.258296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:52.258296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:52.258303 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:52.259776 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:52.260095 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:52.260096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:52.260096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:52.260146 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:52.260325 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:52.260325 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:52.260326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:52.260489 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:52.260578 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:52.260720 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 19:07:52.260784 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:07:52.263268 1382 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:07:52.264967 1382 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:07:52.271934 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.267745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49430 (local address 127.0.203.1:34921)
0620 19:07:52.267919 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:52.267923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:52.268430 (+ 507us) server_negotiation.cc:407] Connection header received
0620 19:07:52.268486 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:52.268489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:52.268507 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:52.268582 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:52.268815 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.269888 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.270842 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.271519 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.271599 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.271661 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:52.271799 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 19:07:52.271848 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260620 19:07:52.271953 1852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.267506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:34921 (local address 127.0.0.1:49430)
0620 19:07:52.268371 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:52.268380 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:52.268426 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:52.268674 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:52.268675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:52.268677 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:52.268723 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.268725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.269972 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.269973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.270729 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.270731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.271610 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.271611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.271687 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.271700 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 19:07:52.271774 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:07:52.271841 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"thread_start_us":106,"threads_started":1}
I20260620 19:07:52.271970 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.267505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:43453 (local address 127.0.0.1:57924)
0620 19:07:52.267994 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:52.268036 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:52.268132 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:52.268503 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:52.268506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:52.268509 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:52.268578 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.268580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.269530 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.269531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.270745 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.270747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.271761 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.271762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.271816 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.271819 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:07:52.271863 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 19:07:52.271895 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":61,"threads_started":1}
I20260620 19:07:52.272372 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.267594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57924 (local address 127.0.203.2:43453)
0620 19:07:52.267756 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:52.267761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:52.268119 (+ 358us) server_negotiation.cc:407] Connection header received
0620 19:07:52.268220 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:52.268223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:52.268239 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:52.268316 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:52.268663 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.269430 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.270839 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.271607 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.271671 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.271729 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:52.272165 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 19:07:52.272306 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260620 19:07:52.272756 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.267785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57100 (local address 127.0.203.3:42777)
0620 19:07:52.267955 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:52.267960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:52.268398 (+ 438us) server_negotiation.cc:407] Connection header received
0620 19:07:52.268442 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:52.268445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:52.268461 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:52.268530 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:52.268699 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.269810 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.271370 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.272133 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.272199 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.272246 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:52.272620 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 19:07:52.272672 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":104,"threads_started":1}
I20260620 19:07:52.272941 1507 tablet_service.cc:1467] Processing CreateTablet for tablet 281fbdf5e55e416087e9c9d22709b7c6 (DEFAULT_TABLE table=test-table [id=2735aa72abf44f9d80128f8564cb6e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:52.273051 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 281fbdf5e55e416087e9c9d22709b7c6 (DEFAULT_TABLE table=test-table [id=2735aa72abf44f9d80128f8564cb6e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:52.273351 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281fbdf5e55e416087e9c9d22709b7c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:52.273350 1507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281fbdf5e55e416087e9c9d22709b7c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:52.275369 1854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.267506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:42777 (local address 127.0.0.1:57100)
0620 19:07:52.268240 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:52.268249 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:52.268294 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:52.268596 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:52.268597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:52.268599 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:52.268642 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.268644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.269918 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.269919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.271281 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.271283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.272280 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.272281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.272361 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.272365 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:07:52.272429 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 19:07:52.272472 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":103,"threads_started":1}
I20260620 19:07:52.276088 1775 tablet_service.cc:1467] Processing CreateTablet for tablet 281fbdf5e55e416087e9c9d22709b7c6 (DEFAULT_TABLE table=test-table [id=2735aa72abf44f9d80128f8564cb6e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:52.276422 1775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281fbdf5e55e416087e9c9d22709b7c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:52.276803 1859 tablet_bootstrap.cc:492] T 281fbdf5e55e416087e9c9d22709b7c6 P 9e5d2a44600240ba8901aaf4051650b2: Bootstrap starting.
I20260620 19:07:52.277518 1859 tablet_bootstrap.cc:654] T 281fbdf5e55e416087e9c9d22709b7c6 P 9e5d2a44600240ba8901aaf4051650b2: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:52.278095 1859 log.cc:824] T 281fbdf5e55e416087e9c9d22709b7c6 P 9e5d2a44600240ba8901aaf4051650b2: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:52.278479 1858 tablet_bootstrap.cc:492] T 281fbdf5e55e416087e9c9d22709b7c6 P 993df3a8d2154f7da1c7eecf236923bf: Bootstrap starting.
I20260620 19:07:52.278947 1859 tablet_bootstrap.cc:492] T 281fbdf5e55e416087e9c9d22709b7c6 P 9e5d2a44600240ba8901aaf4051650b2: No bootstrap required, opened a new log
I20260620 19:07:52.279028 1859 ts_tablet_manager.cc:1397] T 281fbdf5e55e416087e9c9d22709b7c6 P 9e5d2a44600240ba8901aaf4051650b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:07:52.279191 1858 tablet_bootstrap.cc:654] T 281fbdf5e55e416087e9c9d22709b7c6 P 993df3a8d2154f7da1c7eecf236923bf: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:52.279727 1858 log.cc:824] T 281fbdf5e55e416087e9c9d22709b7c6 P 993df3a8d2154f7da1c7eecf236923bf: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:52.280055 1862 tablet_bootstrap.cc:492] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453: Bootstrap starting.
I20260620 19:07:52.280638 1858 tablet_bootstrap.cc:492] T 281fbdf5e55e416087e9c9d22709b7c6 P 993df3a8d2154f7da1c7eecf236923bf: No bootstrap required, opened a new log
I20260620 19:07:52.280726 1858 ts_tablet_manager.cc:1397] T 281fbdf5e55e416087e9c9d22709b7c6 P 993df3a8d2154f7da1c7eecf236923bf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:07:52.281468 1862 tablet_bootstrap.cc:654] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:52.281426 1859 raft_consensus.cc:348] T 281fbdf5e55e416087e9c9d22709b7c6 P 9e5d2a44600240ba8901aaf4051650b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993df3a8d2154f7da1c7eecf236923bf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 43453 } } peers { permanent_uuid: "9e5d2a44600240ba8901aaf4051650b2" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 34921 } } peers { permanent_uuid: "247c68dd5fe44eec9273a46750e5a453" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 42777 } }
I20260620 19:07:52.281637 1859 raft_consensus.cc:374] T 281fbdf5e55e416087e9c9d22709b7c6 P 9e5d2a44600240ba8901aaf4051650b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:52.281685 1859 raft_consensus.cc:729] T 281fbdf5e55e416087e9c9d22709b7c6 P 9e5d2a44600240ba8901aaf4051650b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e5d2a44600240ba8901aaf4051650b2, State: Initialized, Role: FOLLOWER
I20260620 19:07:52.281852 1859 consensus_queue.cc:260] T 281fbdf5e55e416087e9c9d22709b7c6 P 9e5d2a44600240ba8901aaf4051650b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993df3a8d2154f7da1c7eecf236923bf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 43453 } } peers { permanent_uuid: "9e5d2a44600240ba8901aaf4051650b2" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 34921 } } peers { permanent_uuid: "247c68dd5fe44eec9273a46750e5a453" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 42777 } }
I20260620 19:07:52.282033 1862 log.cc:824] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:52.282147 1859 ts_tablet_manager.cc:1428] T 281fbdf5e55e416087e9c9d22709b7c6 P 9e5d2a44600240ba8901aaf4051650b2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 19:07:52.282292 1573 heartbeater.cc:503] Master 127.0.203.62:40793 was elected leader, sending a full tablet report...
I20260620 19:07:52.282564 1858 raft_consensus.cc:348] T 281fbdf5e55e416087e9c9d22709b7c6 P 993df3a8d2154f7da1c7eecf236923bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993df3a8d2154f7da1c7eecf236923bf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 43453 } } peers { permanent_uuid: "9e5d2a44600240ba8901aaf4051650b2" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 34921 } } peers { permanent_uuid: "247c68dd5fe44eec9273a46750e5a453" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 42777 } }
I20260620 19:07:52.282743 1858 raft_consensus.cc:374] T 281fbdf5e55e416087e9c9d22709b7c6 P 993df3a8d2154f7da1c7eecf236923bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:52.282784 1858 raft_consensus.cc:729] T 281fbdf5e55e416087e9c9d22709b7c6 P 993df3a8d2154f7da1c7eecf236923bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993df3a8d2154f7da1c7eecf236923bf, State: Initialized, Role: FOLLOWER
I20260620 19:07:52.282898 1858 consensus_queue.cc:260] T 281fbdf5e55e416087e9c9d22709b7c6 P 993df3a8d2154f7da1c7eecf236923bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993df3a8d2154f7da1c7eecf236923bf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 43453 } } peers { permanent_uuid: "9e5d2a44600240ba8901aaf4051650b2" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 34921 } } peers { permanent_uuid: "247c68dd5fe44eec9273a46750e5a453" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 42777 } }
I20260620 19:07:52.283221 1858 ts_tablet_manager.cc:1428] T 281fbdf5e55e416087e9c9d22709b7c6 P 993df3a8d2154f7da1c7eecf236923bf: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 19:07:52.283237 1707 heartbeater.cc:503] Master 127.0.203.62:40793 was elected leader, sending a full tablet report...
I20260620 19:07:52.283933 1862 tablet_bootstrap.cc:492] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453: No bootstrap required, opened a new log
I20260620 19:07:52.284003 1862 ts_tablet_manager.cc:1397] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 19:07:52.285527 1862 raft_consensus.cc:348] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993df3a8d2154f7da1c7eecf236923bf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 43453 } } peers { permanent_uuid: "9e5d2a44600240ba8901aaf4051650b2" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 34921 } } peers { permanent_uuid: "247c68dd5fe44eec9273a46750e5a453" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 42777 } }
I20260620 19:07:52.285686 1862 raft_consensus.cc:374] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:52.285730 1862 raft_consensus.cc:729] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247c68dd5fe44eec9273a46750e5a453, State: Initialized, Role: FOLLOWER
I20260620 19:07:52.285909 1862 consensus_queue.cc:260] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993df3a8d2154f7da1c7eecf236923bf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 43453 } } peers { permanent_uuid: "9e5d2a44600240ba8901aaf4051650b2" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 34921 } } peers { permanent_uuid: "247c68dd5fe44eec9273a46750e5a453" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 42777 } }
I20260620 19:07:52.286193 1862 ts_tablet_manager.cc:1428] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 19:07:52.286221 1841 heartbeater.cc:503] Master 127.0.203.62:40793 was elected leader, sending a full tablet report...
W20260620 19:07:52.319097 1708 tablet.cc:2369] T 281fbdf5e55e416087e9c9d22709b7c6 P 993df3a8d2154f7da1c7eecf236923bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:07:52.321530 1866 raft_consensus.cc:482] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:07:52.321650 1866 raft_consensus.cc:504] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993df3a8d2154f7da1c7eecf236923bf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 43453 } } peers { permanent_uuid: "9e5d2a44600240ba8901aaf4051650b2" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 34921 } } peers { permanent_uuid: "247c68dd5fe44eec9273a46750e5a453" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 42777 } }
I20260620 19:07:52.322199 1866 leader_election.cc:288] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993df3a8d2154f7da1c7eecf236923bf (127.0.203.2:43453), 9e5d2a44600240ba8901aaf4051650b2 (127.0.203.1:34921)
I20260620 19:07:52.325861 1867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.322425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:34921 (local address 127.0.203.3:38219)
0620 19:07:52.322616 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:52.322627 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:52.322697 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:52.322940 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:52.322940 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:52.322942 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:52.323024 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.323026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.323889 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.323889 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.325036 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.325038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.325586 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.325587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.325667 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.325673 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:07:52.325729 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 19:07:52.325773 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":7,"thread_start_us":104,"threads_started":1}
I20260620 19:07:52.325881 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.322521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:38219 (local address 127.0.203.1:34921)
0620 19:07:52.322569 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:52.322571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:52.322722 (+ 151us) server_negotiation.cc:407] Connection header received
0620 19:07:52.322748 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:52.322749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:52.322758 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:52.322933 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:52.323105 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.323807 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.325112 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.325495 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.325555 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.325564 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:52.325746 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 19:07:52.325791 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:07:52.325986 1843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.322351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:43453 (local address 127.0.203.3:60937)
0620 19:07:52.322409 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:52.322421 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:52.322467 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:52.322649 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:52.322650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:52.322652 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:52.322734 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.322737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.323812 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.323813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.325036 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:52.325038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.325690 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.325691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:52.325802 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.325805 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:07:52.325863 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 19:07:52.325916 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":5}
I20260620 19:07:52.326040 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.322403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:60937 (local address 127.0.203.2:43453)
0620 19:07:52.322447 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:52.322448 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:52.322458 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:07:52.322532 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:52.322533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:52.322546 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:52.322631 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:52.322934 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.323712 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.325169 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.325611 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.325684 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.325696 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:52.325894 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 19:07:52.325957 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:07:52.326282 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281fbdf5e55e416087e9c9d22709b7c6" candidate_uuid: "247c68dd5fe44eec9273a46750e5a453" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993df3a8d2154f7da1c7eecf236923bf" is_pre_election: true
I20260620 19:07:52.326287 1527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281fbdf5e55e416087e9c9d22709b7c6" candidate_uuid: "247c68dd5fe44eec9273a46750e5a453" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e5d2a44600240ba8901aaf4051650b2" is_pre_election: true
I20260620 19:07:52.326431 1661 raft_consensus.cc:2454] T 281fbdf5e55e416087e9c9d22709b7c6 P 993df3a8d2154f7da1c7eecf236923bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 247c68dd5fe44eec9273a46750e5a453 in term 0.
I20260620 19:07:52.326440 1527 raft_consensus.cc:2454] T 281fbdf5e55e416087e9c9d22709b7c6 P 9e5d2a44600240ba8901aaf4051650b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 247c68dd5fe44eec9273a46750e5a453 in term 0.
I20260620 19:07:52.326653 1730 leader_election.cc:302] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247c68dd5fe44eec9273a46750e5a453, 9e5d2a44600240ba8901aaf4051650b2; no voters:
I20260620 19:07:52.326778 1866 raft_consensus.cc:2781] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:07:52.326830 1866 raft_consensus.cc:482] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:07:52.326853 1866 raft_consensus.cc:3037] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:52.327632 1866 raft_consensus.cc:504] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993df3a8d2154f7da1c7eecf236923bf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 43453 } } peers { permanent_uuid: "9e5d2a44600240ba8901aaf4051650b2" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 34921 } } peers { permanent_uuid: "247c68dd5fe44eec9273a46750e5a453" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 42777 } }
I20260620 19:07:52.327900 1866 leader_election.cc:288] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453 [CANDIDATE]: Term 1 election: Requested vote from peers 993df3a8d2154f7da1c7eecf236923bf (127.0.203.2:43453), 9e5d2a44600240ba8901aaf4051650b2 (127.0.203.1:34921)
I20260620 19:07:52.327957 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281fbdf5e55e416087e9c9d22709b7c6" candidate_uuid: "247c68dd5fe44eec9273a46750e5a453" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993df3a8d2154f7da1c7eecf236923bf"
I20260620 19:07:52.328060 1661 raft_consensus.cc:3037] T 281fbdf5e55e416087e9c9d22709b7c6 P 993df3a8d2154f7da1c7eecf236923bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:52.328104 1527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281fbdf5e55e416087e9c9d22709b7c6" candidate_uuid: "247c68dd5fe44eec9273a46750e5a453" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e5d2a44600240ba8901aaf4051650b2"
I20260620 19:07:52.328183 1527 raft_consensus.cc:3037] T 281fbdf5e55e416087e9c9d22709b7c6 P 9e5d2a44600240ba8901aaf4051650b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:52.329043 1661 raft_consensus.cc:2454] T 281fbdf5e55e416087e9c9d22709b7c6 P 993df3a8d2154f7da1c7eecf236923bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 247c68dd5fe44eec9273a46750e5a453 in term 1.
I20260620 19:07:52.329123 1527 raft_consensus.cc:2454] T 281fbdf5e55e416087e9c9d22709b7c6 P 9e5d2a44600240ba8901aaf4051650b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 247c68dd5fe44eec9273a46750e5a453 in term 1.
I20260620 19:07:52.329247 1728 leader_election.cc:302] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247c68dd5fe44eec9273a46750e5a453, 993df3a8d2154f7da1c7eecf236923bf; no voters:
I20260620 19:07:52.329363 1866 raft_consensus.cc:2781] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:07:52.329516 1866 raft_consensus.cc:686] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453 [term 1 LEADER]: Becoming Leader. State: Replica: 247c68dd5fe44eec9273a46750e5a453, State: Running, Role: LEADER
I20260620 19:07:52.329598 1866 consensus_queue.cc:237] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993df3a8d2154f7da1c7eecf236923bf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 43453 } } peers { permanent_uuid: "9e5d2a44600240ba8901aaf4051650b2" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 34921 } } peers { permanent_uuid: "247c68dd5fe44eec9273a46750e5a453" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 42777 } }
I20260620 19:07:52.330444 1381 catalog_manager.cc:5495] T 281fbdf5e55e416087e9c9d22709b7c6 P 247c68dd5fe44eec9273a46750e5a453 reported cstate change: term changed from 0 to 1, leader changed from <none> to 247c68dd5fe44eec9273a46750e5a453 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "247c68dd5fe44eec9273a46750e5a453" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993df3a8d2154f7da1c7eecf236923bf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 43453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e5d2a44600240ba8901aaf4051650b2" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "247c68dd5fe44eec9273a46750e5a453" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 42777 } health_report { overall_health: HEALTHY } } }
Jun 20 19:07:52 dist-test-slave-wd30 krb5kdc[1337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.1@KRBTEST.COM
I20260620 19:07:52.349195 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.341087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49436 (local address 127.0.203.1:34921)
0620 19:07:52.341180 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:52.341183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:52.341264 (+ 81us) server_negotiation.cc:407] Connection header received
0620 19:07:52.341338 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:52.341340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:52.341353 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:52.341432 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:52.342038 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.342528 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.343320 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.343512 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.343587 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.346087 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:52.346101 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:52.346102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:52.346118 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:52.347939 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:52.348311 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:52.348315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:52.348315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:52.348380 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:52.348596 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:52.348597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:52.348598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:52.348762 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:52.348919 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:52.348990 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 19:07:52.349075 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 19:07:52 dist-test-slave-wd30 krb5kdc[1337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982472, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:07:52 dist-test-slave-wd30 krb5kdc[1337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982472, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.203.1@KRBTEST.COM
I20260620 19:07:52.373251 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.365495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49442 (local address 127.0.203.1:34921)
0620 19:07:52.365567 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:52.365569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:52.365670 (+ 101us) server_negotiation.cc:407] Connection header received
0620 19:07:52.365754 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:52.365754 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:52.365773 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:52.365852 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:52.366516 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.367031 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.367850 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.368051 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.368162 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.370587 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:52.370589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:52.370589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:52.370597 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:52.372017 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:52.372390 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:52.372391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:52.372391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:52.372441 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:52.372660 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:52.372661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:52.372661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:52.372750 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:52.372822 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:52.373026 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 19:07:52.373115 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 19:07:52 dist-test-slave-wd30 krb5kdc[1337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:07:52 dist-test-slave-wd30 krb5kdc[1337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.1@KRBTEST.COM
I20260620 19:07:52.396561 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:52.389274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49444 (local address 127.0.203.1:34921)
0620 19:07:52.389390 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:52.389393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:52.389502 (+ 109us) server_negotiation.cc:407] Connection header received
0620 19:07:52.389656 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:52.389657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:52.389668 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:52.389744 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:52.390317 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.390799 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.391736 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:52.391929 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:52.392035 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:52.394316 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:52.394318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:52.394318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:52.394326 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:52.395598 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:52.395920 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:52.395921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:52.395922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:52.395976 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:52.396138 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:52.396139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:52.396139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:52.396225 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:52.396300 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:52.396433 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 19:07:52.396491 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 19:07:52.397177 1487 tablet_service.cc:2908] Rejecting scan request for tablet 281fbdf5e55e416087e9c9d22709b7c6: Uninitialized: clean time has not yet been initialized
I20260620 19:07:52.398052 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 1444
I20260620 19:07:52.403307 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 1578
I20260620 19:07:52.407847 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 1712
I20260620 19:07:52.412210 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 1350
2026-06-20T19:07:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2029 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:07:52 dist-test-slave-wd30 krb5kdc[1882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:07:52 dist-test-slave-wd30 krb5kdc[1882](info): set up 2 sockets
Jun 20 19:07:52 dist-test-slave-wd30 krb5kdc[1882](info): commencing operation
krb5kdc: starting...
W20260620 19:07:53.450773 812 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:07:53 dist-test-slave-wd30 krb5kdc[1882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:07:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:07:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:07:53.569661 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:42077
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:42077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:53.652245 1895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:53.652472 1895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:53.652496 1895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:53.653678 1895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:07:53.653726 1895 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 19:07:53.653740 1895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:53.653754 1895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:53.653767 1895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:07:53.653779 1895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:07:53.655313 1895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:42077
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:42077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:07:53.655509 1895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:53.655701 1895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:53.660899 1901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:53.660987 1895 server_base.cc:1034] running on GCE node
W20260620 19:07:53.660926 1900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:53.660926 1903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:53.661337 1895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:53.661680 1895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:53.662858 1895 hybrid_clock.cc:648] HybridClock initialized: now 1781982473662838 us; error 44 us; skew 500 ppm
Jun 20 19:07:53 dist-test-slave-wd30 krb5kdc[1882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:53.668134 1895 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:07:53.668622 1895 webserver.cc:460] Webserver started at http://127.0.203.62:41555/ using document root <none> and password file <none>
I20260620 19:07:53.668833 1895 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:53.668874 1895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:53.668977 1895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:53.669987 1895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "fc0b0b475a914ac3b4f2b1bd5035435e"
format_stamp: "Formatted at 2026-06-20 19:07:53 on dist-test-slave-wd30"
server_key: "f861417d25d59ede2adcc18d198bed55"
server_key_iv: "1d1d17559a8dadd962595048db994cee"
server_key_version: "encryptionkey@0"
I20260620 19:07:53.670288 1895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "fc0b0b475a914ac3b4f2b1bd5035435e"
format_stamp: "Formatted at 2026-06-20 19:07:53 on dist-test-slave-wd30"
server_key: "f861417d25d59ede2adcc18d198bed55"
server_key_iv: "1d1d17559a8dadd962595048db994cee"
server_key_version: "encryptionkey@0"
I20260620 19:07:53.671622 1895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:07:53.672354 1910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:53.672528 1895 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:07:53.672623 1895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "fc0b0b475a914ac3b4f2b1bd5035435e"
format_stamp: "Formatted at 2026-06-20 19:07:53 on dist-test-slave-wd30"
server_key: "f861417d25d59ede2adcc18d198bed55"
server_key_iv: "1d1d17559a8dadd962595048db994cee"
server_key_version: "encryptionkey@0"
I20260620 19:07:53.672684 1895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:53.687991 1895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:53.690199 1895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:53.690308 1895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:53.694823 1895 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:42077
I20260620 19:07:53.694836 1972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:42077 every 8 connection(s)
I20260620 19:07:53.695313 1895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:07:53.696070 1973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:53.698594 1973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e: Bootstrap starting.
I20260620 19:07:53.699726 1973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:53.700155 1973 log.cc:824] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:53.701077 1973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e: No bootstrap required, opened a new log
I20260620 19:07:53.703416 1973 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0b0b475a914ac3b4f2b1bd5035435e" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42077 } }
I20260620 19:07:53.703616 1973 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:53.703670 1973 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc0b0b475a914ac3b4f2b1bd5035435e, State: Initialized, Role: FOLLOWER
I20260620 19:07:53.703850 1973 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0b0b475a914ac3b4f2b1bd5035435e" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42077 } }
I20260620 19:07:53.703938 1973 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:07:53.703984 1973 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:07:53.704035 1973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:53.704664 1973 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0b0b475a914ac3b4f2b1bd5035435e" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42077 } }
I20260620 19:07:53.704792 1973 leader_election.cc:302] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc0b0b475a914ac3b4f2b1bd5035435e; no voters:
I20260620 19:07:53.705003 1973 leader_election.cc:288] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:07:53.705049 1976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:07:53.705294 1973 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:07:53.705298 1976 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e [term 1 LEADER]: Becoming Leader. State: Replica: fc0b0b475a914ac3b4f2b1bd5035435e, State: Running, Role: LEADER
I20260620 19:07:53.705413 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 1895
I20260620 19:07:53.705497 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:07:53.705490 1976 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0b0b475a914ac3b4f2b1bd5035435e" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42077 } }
I20260620 19:07:53.705674 812 external_mini_cluster.cc:1428] Setting key d24b6b570fffb4f400f6eba733a1c77f
I20260620 19:07:53.705863 1977 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc0b0b475a914ac3b4f2b1bd5035435e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0b0b475a914ac3b4f2b1bd5035435e" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42077 } } }
I20260620 19:07:53.705895 1978 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc0b0b475a914ac3b4f2b1bd5035435e. Latest consensus state: current_term: 1 leader_uuid: "fc0b0b475a914ac3b4f2b1bd5035435e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0b0b475a914ac3b4f2b1bd5035435e" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42077 } } }
I20260620 19:07:53.705950 1977 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e [sys.catalog]: This master's current role is: LEADER
I20260620 19:07:53.705958 1978 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e [sys.catalog]: This master's current role is: LEADER
I20260620 19:07:53.706375 1982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:07:53.706959 1982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:07:53.709059 1982 catalog_manager.cc:1347] Generated new cluster ID: 6d09df061249469ca6565cc8c30eba1e
I20260620 19:07:53.709115 1982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:07:53 dist-test-slave-wd30 krb5kdc[1882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:07:53.725689 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:53.706343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.0.203.62:42077)
0620 19:07:53.707728 (+ 1385us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:53.707736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:53.707763 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:07:53.707869 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:53.707875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:53.707927 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:53.708048 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:53.708801 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:53.716525 (+ 7724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:53.717730 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:53.718389 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:53.718506 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:53.721561 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:53.721567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:53.721568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:53.721583 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:53.723708 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:53.724124 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:53.724128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:53.724128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:53.724286 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:53.724499 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:53.724500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:53.724500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:53.724857 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:53.725061 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:53.725152 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 19:07:53.725433 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1342,"thread_start_us":102,"threads_started":1}
I20260620 19:07:53.731353 1982 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:07:53.731971 1982 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:07:53.739539 1982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e: Generated new TSK 0
I20260620 19:07:53.739774 1982 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
I20260620 19:07:53.771389 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:42077
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:53.854277 1999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:53.854518 1999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:53.854545 1999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:53.855825 1999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:53.855871 1999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:53.855893 1999 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:07:53.855917 1999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:07:53.857570 1999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.203.62:42077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:07:53.857779 1999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:53.857975 1999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:53.863027 2004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:53.863039 2007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:53.863027 2005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:53.863325 1999 server_base.cc:1034] running on GCE node
I20260620 19:07:53.863541 1999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:53.863726 1999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:53.864905 1999 hybrid_clock.cc:648] HybridClock initialized: now 1781982473864885 us; error 38 us; skew 500 ppm
Jun 20 19:07:53 dist-test-slave-wd30 krb5kdc[1882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:53.867570 1999 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:07:53.868026 1999 webserver.cc:460] Webserver started at http://127.0.203.1:33811/ using document root <none> and password file <none>
I20260620 19:07:53.868257 1999 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:53.868322 1999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:53.868477 1999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:53.869517 1999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "a57b03d0719b40d5b05ce328f16736a8"
format_stamp: "Formatted at 2026-06-20 19:07:53 on dist-test-slave-wd30"
server_key: "1740914fc79ec21b4a1db6e3ebfdfcb1"
server_key_iv: "052faa51ec7bbcec1d010b9993765aee"
server_key_version: "encryptionkey@0"
I20260620 19:07:53.869874 1999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "a57b03d0719b40d5b05ce328f16736a8"
format_stamp: "Formatted at 2026-06-20 19:07:53 on dist-test-slave-wd30"
server_key: "1740914fc79ec21b4a1db6e3ebfdfcb1"
server_key_iv: "052faa51ec7bbcec1d010b9993765aee"
server_key_version: "encryptionkey@0"
I20260620 19:07:53.871248 1999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:07:53.872032 2014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:53.872215 1999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:07:53.872311 1999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "a57b03d0719b40d5b05ce328f16736a8"
format_stamp: "Formatted at 2026-06-20 19:07:53 on dist-test-slave-wd30"
server_key: "1740914fc79ec21b4a1db6e3ebfdfcb1"
server_key_iv: "052faa51ec7bbcec1d010b9993765aee"
server_key_version: "encryptionkey@0"
I20260620 19:07:53.872396 1999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:53.881386 1999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:53.883682 1999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:53.883836 1999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:53.884295 1999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:53.884358 1999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:53.884400 1999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:53.884424 1999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:07:53 dist-test-slave-wd30 krb5kdc[1882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:07:53.891812 1999 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:39453
I20260620 19:07:53.891891 2129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:39453 every 8 connection(s)
I20260620 19:07:53.892184 1999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:07:53.894616 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:53.884803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:44825 (local address 127.0.203.62:42077)
0620 19:07:53.884873 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:53.884876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:53.885075 (+ 199us) server_negotiation.cc:407] Connection header received
0620 19:07:53.885517 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:53.885519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:53.885536 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:53.885626 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:53.886376 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:53.887290 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:53.888312 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:53.888531 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:53.888619 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:53.891959 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:53.891961 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:53.891961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:53.891969 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:53.893498 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:53.893905 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:53.893906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:53.893906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:53.893958 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:53.894152 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:53.894152 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:53.894152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:53.894251 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:53.894300 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:53.894484 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 19:07:53.894553 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:07:53.894824 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:53.884769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42077 (local address 127.0.203.1:44825)
0620 19:07:53.884923 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:53.884962 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:53.885417 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:53.885673 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:53.885675 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:53.885971 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:53.886262 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:53.886267 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:53.887387 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:53.887388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:53.888213 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:53.888216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:53.888631 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:53.888632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:53.888761 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:53.889278 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:53.889298 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:53.891838 (+ 2540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:53.893598 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:53.893604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:53.893605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:53.893821 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:53.894013 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:53.894013 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:53.894014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:53.894100 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:53.894289 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:53.894291 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:53.894398 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 19:07:53.894471 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 19:07:53.894584 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260620 19:07:53.895359 2130 heartbeater.cc:347] Connected to a master server at 127.0.203.62:42077
I20260620 19:07:53.895448 2130 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:53.895642 2130 heartbeater.cc:511] Master 127.0.203.62:42077 requested a full tablet report, sending...
I20260620 19:07:53.896059 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 1999
I20260620 19:07:53.896152 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:07:53.896217 1926 ts_manager.cc:194] Registered new tserver with Master: a57b03d0719b40d5b05ce328f16736a8 (127.0.203.1:39453)
I20260620 19:07:53.896328 812 external_mini_cluster.cc:1428] Setting key 3d6abb65edb4e83160379cc9c1d7d69b
I20260620 19:07:53.897120 1926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:44825
I20260620 19:07:53.902879 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:53.896462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:56515 (local address 127.0.203.62:42077)
0620 19:07:53.897340 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:53.897342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:53.897353 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:07:53.897373 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:53.897374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:53.897385 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:53.897442 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:53.897790 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:53.898265 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:53.898996 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:53.899275 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:53.899358 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:53.900229 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:53.900231 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:53.900231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:53.900238 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:53.901671 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:53.902070 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:53.902071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:53.902071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:53.902130 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:53.902344 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:53.902346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:53.902346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:53.902463 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:53.902542 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:53.902697 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 19:07:53.902781 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854}
I20260620 19:07:53.902894 2136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:53.896411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42077 (local address 127.0.203.1:56515)
0620 19:07:53.896650 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:53.896662 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:53.896709 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:53.897448 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:53.897449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:53.897644 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:53.897706 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:53.897708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:53.898368 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:53.898369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:53.898909 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:53.898910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:53.899365 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:53.899366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:53.899432 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:53.899867 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:53.899871 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:53.900138 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:53.901788 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:53.901789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:53.901789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:53.901976 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:53.902207 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:53.902208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:53.902208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:53.902270 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:53.902534 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:53.902535 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:53.902584 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 19:07:53.902672 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 19:07:53.902783 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for kudu/127.0.203.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
I20260620 19:07:53.922461 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:42077
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:54.005230 2139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:54.005539 2139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:54.005568 2139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:54.007020 2139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:54.007097 2139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:54.007153 2139 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:07:54.007193 2139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:07:54.008952 2139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.203.62:42077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:07:54.009248 2139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:54.009509 2139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:54.014700 2144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:54.014761 2145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:54.014751 2147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:54.014943 2139 server_base.cc:1034] running on GCE node
I20260620 19:07:54.015223 2139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:54.015517 2139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:54.016678 2139 hybrid_clock.cc:648] HybridClock initialized: now 1781982474016666 us; error 29 us; skew 500 ppm
Jun 20 19:07:54 dist-test-slave-wd30 krb5kdc[1882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:54.019376 2139 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:07:54.019824 2139 webserver.cc:460] Webserver started at http://127.0.203.2:34469/ using document root <none> and password file <none>
I20260620 19:07:54.020069 2139 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:54.020116 2139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:54.020263 2139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:54.021347 2139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "e47b3e7139db49d48c3f64034a496f05"
format_stamp: "Formatted at 2026-06-20 19:07:54 on dist-test-slave-wd30"
server_key: "fe7f7728463274c7e00124b08e33f925"
server_key_iv: "0c3ba3b568b25d57982ccdf385600b71"
server_key_version: "encryptionkey@0"
I20260620 19:07:54.021723 2139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "e47b3e7139db49d48c3f64034a496f05"
format_stamp: "Formatted at 2026-06-20 19:07:54 on dist-test-slave-wd30"
server_key: "fe7f7728463274c7e00124b08e33f925"
server_key_iv: "0c3ba3b568b25d57982ccdf385600b71"
server_key_version: "encryptionkey@0"
I20260620 19:07:54.023093 2139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:07:54.023952 2154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:54.024204 2139 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:07:54.024312 2139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "e47b3e7139db49d48c3f64034a496f05"
format_stamp: "Formatted at 2026-06-20 19:07:54 on dist-test-slave-wd30"
server_key: "fe7f7728463274c7e00124b08e33f925"
server_key_iv: "0c3ba3b568b25d57982ccdf385600b71"
server_key_version: "encryptionkey@0"
I20260620 19:07:54.024399 2139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:54.036101 2139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:54.038602 2139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:54.038784 2139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:54.039465 2139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:54.039526 2139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:54.039575 2139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:54.039604 2139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:54.046525 2139 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:40729
I20260620 19:07:54.046657 2269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:40729 every 8 connection(s)
Jun 20 19:07:54 dist-test-slave-wd30 krb5kdc[1882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:07:54.046960 2139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:07:54.047778 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 2139
I20260620 19:07:54.047888 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:07:54.048056 812 external_mini_cluster.cc:1428] Setting key d4555d026c185eedca2b0e9aa419d30f
I20260620 19:07:54.050741 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.039896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:33001 (local address 127.0.203.62:42077)
0620 19:07:54.039968 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.039971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.040209 (+ 238us) server_negotiation.cc:407] Connection header received
0620 19:07:54.040710 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.040712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.040727 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.040815 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:54.041542 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.042379 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.044050 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.044379 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.044450 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.047371 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:54.047381 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:54.047382 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:54.047390 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:54.049229 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:54.049665 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:54.049666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:54.049667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:54.049743 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:54.049991 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:54.049991 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:54.049992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:54.050149 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:54.050247 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.050597 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.050674 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:07:54.051025 2163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.039896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42077 (local address 127.0.203.2:33001)
0620 19:07:54.040062 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:54.040103 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:54.040604 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:54.040906 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:54.040911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:54.041224 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:54.041427 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.041433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.042477 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.042478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.043962 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.043966 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.044457 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.044458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.044601 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.045286 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:54.045296 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:54.047218 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:54.049324 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:54.049331 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:54.049333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:54.049586 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:54.049812 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:54.049813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:54.049813 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:54.049927 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:54.050227 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:54.050233 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:54.050419 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 19:07:54.050547 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 19:07:54.050735 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260620 19:07:54.051559 2270 heartbeater.cc:347] Connected to a master server at 127.0.203.62:42077
I20260620 19:07:54.051651 2270 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:54.051841 2270 heartbeater.cc:511] Master 127.0.203.62:42077 requested a full tablet report, sending...
I20260620 19:07:54.052207 1927 ts_manager.cc:194] Registered new tserver with Master: e47b3e7139db49d48c3f64034a496f05 (127.0.203.2:40729)
I20260620 19:07:54.052757 1927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:33001
WARNING: no policy specified for kudu/127.0.203.3@KRBTEST.COM; defaulting to no policy
I20260620 19:07:54.058599 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.052753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:51717 (local address 127.0.203.62:42077)
0620 19:07:54.052795 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.052798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.052938 (+ 140us) server_negotiation.cc:407] Connection header received
0620 19:07:54.053013 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.053014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.053024 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.053065 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:54.053404 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.053884 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.054609 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.054818 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.054904 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.055895 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:54.055897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:54.055897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:54.055905 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:54.057404 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:54.057745 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:54.057746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:54.057747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:54.057804 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:54.058016 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:54.058016 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:54.058017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:54.058173 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:54.058256 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.058409 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.058497 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:07:54.058614 2277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.052712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42077 (local address 127.0.203.2:51717)
0620 19:07:54.052882 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:54.052892 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:54.052962 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:54.053079 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:54.053079 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:54.053259 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:54.053317 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.053318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.053984 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.053985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.054524 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.054525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.054906 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.054907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.054972 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.055513 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:54.055518 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:54.055802 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:54.057481 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:54.057482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:54.057482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:54.057656 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:54.057880 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:54.057881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:54.057881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:54.057942 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:54.058247 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:54.058248 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:54.058296 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 19:07:54.058385 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 19:07:54.058501 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":63,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
I20260620 19:07:54.074438 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:42077
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:54.158747 2279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:54.159024 2279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:54.159068 2279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:54.160279 2279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:54.160356 2279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:54.160394 2279 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:07:54.160439 2279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:07:54.162019 2279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.203.62:42077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:07:54.162304 2279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:54.162550 2279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:54.167670 2285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:54.167675 2279 server_base.cc:1034] running on GCE node
W20260620 19:07:54.167676 2284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:54.167709 2287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:54.168133 2279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:54.168386 2279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:54.169546 2279 hybrid_clock.cc:648] HybridClock initialized: now 1781982474169536 us; error 30 us; skew 500 ppm
Jun 20 19:07:54 dist-test-slave-wd30 krb5kdc[1882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:54.172013 2279 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:07:54.172444 2279 webserver.cc:460] Webserver started at http://127.0.203.3:43003/ using document root <none> and password file <none>
I20260620 19:07:54.172626 2279 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:54.172667 2279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:54.172749 2279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:54.173709 2279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "78739bdb24214422921dc8329589abf7"
format_stamp: "Formatted at 2026-06-20 19:07:54 on dist-test-slave-wd30"
server_key: "c80c2714b73aa8c06a582f7367abd8bf"
server_key_iv: "e2f1ec4b7e556ad9e8f72b352dc6a7de"
server_key_version: "encryptionkey@0"
I20260620 19:07:54.174000 2279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "78739bdb24214422921dc8329589abf7"
format_stamp: "Formatted at 2026-06-20 19:07:54 on dist-test-slave-wd30"
server_key: "c80c2714b73aa8c06a582f7367abd8bf"
server_key_iv: "e2f1ec4b7e556ad9e8f72b352dc6a7de"
server_key_version: "encryptionkey@0"
I20260620 19:07:54.175441 2279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:07:54.176203 2294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:54.176400 2279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:07:54.176505 2279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "78739bdb24214422921dc8329589abf7"
format_stamp: "Formatted at 2026-06-20 19:07:54 on dist-test-slave-wd30"
server_key: "c80c2714b73aa8c06a582f7367abd8bf"
server_key_iv: "e2f1ec4b7e556ad9e8f72b352dc6a7de"
server_key_version: "encryptionkey@0"
I20260620 19:07:54.176594 2279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:54.191706 2279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:54.193399 2279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:54.193568 2279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:54.194043 2279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:54.194105 2279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:54.194168 2279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:54.194199 2279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:54.201326 2279 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:44943
I20260620 19:07:54.201368 2409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:44943 every 8 connection(s)
I20260620 19:07:54.201700 2279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:07:54 dist-test-slave-wd30 krb5kdc[1882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:07:54.205214 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.194587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:41001 (local address 127.0.203.62:42077)
0620 19:07:54.194662 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.194665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.194872 (+ 207us) server_negotiation.cc:407] Connection header received
0620 19:07:54.195467 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.195469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.195483 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.195564 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:54.196401 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.197239 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.198967 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.199302 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.199367 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.202290 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:54.202291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:54.202292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:54.202299 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:54.203778 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:54.204208 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:54.204209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:54.204210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:54.204274 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:54.204534 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:54.204535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:54.204536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:54.204656 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:54.204734 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.205037 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.205121 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:07:54.205425 2303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.194585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42077 (local address 127.0.203.3:41001)
0620 19:07:54.194743 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:54.194783 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:54.195368 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:54.195636 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:54.195639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:54.196011 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:54.196308 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.196315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.197346 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.197347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.198858 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.198861 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.199394 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.199395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.199575 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.200303 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:54.200316 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:54.202187 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:54.203883 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:54.203890 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:54.203892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:54.204126 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:54.204343 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:54.204343 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:54.204344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:54.204466 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:54.204724 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:54.204730 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:54.204878 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 19:07:54.205009 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 19:07:54.205177 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
I20260620 19:07:54.206032 2410 heartbeater.cc:347] Connected to a master server at 127.0.203.62:42077
I20260620 19:07:54.206136 2410 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:54.206355 2410 heartbeater.cc:511] Master 127.0.203.62:42077 requested a full tablet report, sending...
I20260620 19:07:54.206795 1926 ts_manager.cc:194] Registered new tserver with Master: 78739bdb24214422921dc8329589abf7 (127.0.203.3:44943)
I20260620 19:07:54.207371 1926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:41001
I20260620 19:07:54.209590 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 2279
I20260620 19:07:54.209710 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:07:54.209872 812 external_mini_cluster.cc:1428] Setting key e2260d3e9d1082ea407205594d81f295
I20260620 19:07:54.210789 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:07:54.213186 2416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.207199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42077 (local address 127.0.203.3:40089)
0620 19:07:54.207622 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:54.207634 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:54.207687 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:54.207815 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:54.207816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:54.208002 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:54.208055 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.208057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.208708 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.208709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.209235 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.209237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.209604 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.209604 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.209667 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.210194 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:54.210199 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:54.210593 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:54.212122 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:54.212123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:54.212124 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:54.212362 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:54.212549 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:54.212550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:54.212550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:54.212636 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:54.212940 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:54.212941 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:54.212985 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 19:07:54.213038 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 19:07:54.213120 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":290,"threads_started":1}
I20260620 19:07:54.213279 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.207238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:40089 (local address 127.0.203.62:42077)
0620 19:07:54.207276 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.207278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.207673 (+ 395us) server_negotiation.cc:407] Connection header received
0620 19:07:54.207744 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.207745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.207756 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.207809 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:54.208142 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.208615 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.209320 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.209531 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.209605 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.210683 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:54.210684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:54.210685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:54.210691 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:54.212044 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:54.212435 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:54.212436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:54.212436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:54.212485 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:54.212718 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:54.212719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:54.212719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:54.212868 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:54.212944 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.213082 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.213163 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
Jun 20 19:07:54 dist-test-slave-wd30 krb5kdc[1882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982474, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:07:54 dist-test-slave-wd30 krb5kdc[1882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982474, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:07:54.248211 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.240164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46462 (local address 127.0.203.62:42077)
0620 19:07:54.240223 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.240226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.240241 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:07:54.240278 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.240279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.240294 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.240386 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:54.241025 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.241828 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.242609 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.242823 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.242904 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.245908 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:54.245909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:54.245910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:54.245917 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:54.247236 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:54.247601 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:54.247602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:54.247602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:54.247653 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:54.247818 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:54.247818 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:54.247819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:54.247920 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:54.248008 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.248082 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.248143 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:07:54.249907 1926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:07:54.251715 1926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:07:54.258455 2431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.254304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39572 (local address 127.0.203.2:40729)
0620 19:07:54.254489 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.254494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.254823 (+ 329us) server_negotiation.cc:407] Connection header received
0620 19:07:54.254866 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.254869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.254884 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.254957 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:54.255169 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.256225 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.257327 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.257994 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.258091 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.258170 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.258303 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.258346 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":87,"threads_started":1}
I20260620 19:07:54.258445 2427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.253996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:44943 (local address 127.0.0.1:47870)
0620 19:07:54.254115 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:54.254158 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:54.254223 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:54.254710 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:54.254715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:54.254717 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:54.254797 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.254799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.256166 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.256167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.257257 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.257260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.258179 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.258180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.258251 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.258257 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:07:54.258288 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 19:07:54.258366 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":53,"thread_start_us":66,"threads_started":1}
I20260620 19:07:54.258590 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.254173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47870 (local address 127.0.203.3:44943)
0620 19:07:54.254519 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.254524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.254537 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:07:54.254581 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.254583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.254600 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.254684 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:54.254883 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.256071 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.257362 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.258077 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.258341 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.258413 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.258438 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.258480 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":89,"threads_started":1}
I20260620 19:07:54.258649 2430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.254206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:40729 (local address 127.0.0.1:39572)
0620 19:07:54.254719 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:54.254729 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:54.254769 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:54.255029 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:54.255030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:54.255032 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:54.255072 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.255073 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.256322 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.256323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.257237 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.257239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.258090 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.258091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.258166 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.258181 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 19:07:54.258284 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 19:07:54.258572 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":7,"thread_start_us":75,"threads_started":1}
I20260620 19:07:54.258859 2428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.254045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:39453 (local address 127.0.0.1:51042)
0620 19:07:54.254557 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:54.254567 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:54.254627 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:54.254937 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:54.254938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:54.254941 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:54.255019 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.255021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.255916 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.255917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.257260 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.257261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.258392 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.258393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.258453 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.258455 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:07:54.258779 (+ 324us) client_negotiation.cc:240] Negotiation successful
0620 19:07:54.258814 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":3,"thread_start_us":89,"threads_started":1}
I20260620 19:07:54.259176 2432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.254366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51042 (local address 127.0.203.1:39453)
0620 19:07:54.254572 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.254575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.254619 (+ 44us) server_negotiation.cc:407] Connection header received
0620 19:07:54.254768 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.254770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.254788 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.254863 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:54.255112 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.255825 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.257365 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.258218 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.258279 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.258360 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.258987 (+ 627us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.259037 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":98,"threads_started":1}
I20260620 19:07:54.259595 2344 tablet_service.cc:1467] Processing CreateTablet for tablet eeed0cd09f1c4f9eb2884642e4cc6f92 (DEFAULT_TABLE table=test-table [id=af3b2e0261364840a0237438452b44b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:54.259831 2204 tablet_service.cc:1467] Processing CreateTablet for tablet eeed0cd09f1c4f9eb2884642e4cc6f92 (DEFAULT_TABLE table=test-table [id=af3b2e0261364840a0237438452b44b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:54.259807 2064 tablet_service.cc:1467] Processing CreateTablet for tablet eeed0cd09f1c4f9eb2884642e4cc6f92 (DEFAULT_TABLE table=test-table [id=af3b2e0261364840a0237438452b44b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:54.260030 2344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeed0cd09f1c4f9eb2884642e4cc6f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:54.260146 2204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeed0cd09f1c4f9eb2884642e4cc6f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:54.260146 2064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeed0cd09f1c4f9eb2884642e4cc6f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:54.262619 2433 tablet_bootstrap.cc:492] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7: Bootstrap starting.
I20260620 19:07:54.263800 2433 tablet_bootstrap.cc:654] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:54.264178 2434 tablet_bootstrap.cc:492] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8: Bootstrap starting.
I20260620 19:07:54.264190 2433 log.cc:824] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:54.264770 2434 tablet_bootstrap.cc:654] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:54.264955 2433 tablet_bootstrap.cc:492] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7: No bootstrap required, opened a new log
I20260620 19:07:54.265053 2433 ts_tablet_manager.cc:1397] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:07:54.265098 2434 log.cc:824] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:54.265379 2435 tablet_bootstrap.cc:492] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05: Bootstrap starting.
I20260620 19:07:54.265944 2435 tablet_bootstrap.cc:654] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:54.266247 2435 log.cc:824] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:54.266703 2434 tablet_bootstrap.cc:492] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8: No bootstrap required, opened a new log
I20260620 19:07:54.266775 2434 ts_tablet_manager.cc:1397] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 19:07:54.266994 2435 tablet_bootstrap.cc:492] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05: No bootstrap required, opened a new log
I20260620 19:07:54.267060 2435 ts_tablet_manager.cc:1397] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:07:54.267112 2433 raft_consensus.cc:348] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } }
I20260620 19:07:54.267310 2433 raft_consensus.cc:374] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:54.267352 2433 raft_consensus.cc:729] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78739bdb24214422921dc8329589abf7, State: Initialized, Role: FOLLOWER
I20260620 19:07:54.267475 2433 consensus_queue.cc:260] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } }
I20260620 19:07:54.267738 2433 ts_tablet_manager.cc:1428] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260620 19:07:54.267864 2410 heartbeater.cc:503] Master 127.0.203.62:42077 was elected leader, sending a full tablet report...
I20260620 19:07:54.268760 2434 raft_consensus.cc:348] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } }
I20260620 19:07:54.268923 2434 raft_consensus.cc:374] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:54.268958 2434 raft_consensus.cc:729] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57b03d0719b40d5b05ce328f16736a8, State: Initialized, Role: FOLLOWER
I20260620 19:07:54.269080 2434 consensus_queue.cc:260] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } }
I20260620 19:07:54.269034 2435 raft_consensus.cc:348] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } }
I20260620 19:07:54.269160 2435 raft_consensus.cc:374] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:54.269210 2435 raft_consensus.cc:729] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47b3e7139db49d48c3f64034a496f05, State: Initialized, Role: FOLLOWER
I20260620 19:07:54.269348 2130 heartbeater.cc:503] Master 127.0.203.62:42077 was elected leader, sending a full tablet report...
I20260620 19:07:54.269346 2435 consensus_queue.cc:260] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } }
I20260620 19:07:54.269367 2434 ts_tablet_manager.cc:1428] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:07:54.269615 2270 heartbeater.cc:503] Master 127.0.203.62:42077 was elected leader, sending a full tablet report...
I20260620 19:07:54.269742 2435 ts_tablet_manager.cc:1428] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260620 19:07:54.297612 2271 tablet.cc:2369] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:07:54.303540 2440 raft_consensus.cc:482] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:07:54.303716 2440 raft_consensus.cc:504] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } }
I20260620 19:07:54.304080 2440 leader_election.cc:288] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78739bdb24214422921dc8329589abf7 (127.0.203.3:44943), e47b3e7139db49d48c3f64034a496f05 (127.0.203.2:40729)
I20260620 19:07:54.308274 2431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.304565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:40565 (local address 127.0.203.2:40729)
0620 19:07:54.304633 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.304635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.304717 (+ 82us) server_negotiation.cc:407] Connection header received
0620 19:07:54.304780 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.304781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.304793 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.304848 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:54.304972 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.305670 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.307253 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.307871 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.307950 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.307960 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.308156 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.308198 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:07:54.308274 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.304393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:46529 (local address 127.0.203.3:44943)
0620 19:07:54.304490 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.304492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.304502 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:07:54.304529 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.304530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.304541 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.304632 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:54.304818 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.305631 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.307253 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.307871 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.307950 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.307960 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.308156 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.308199 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 19:07:54.308308 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.304441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:40729 (local address 127.0.203.1:40565)
0620 19:07:54.304635 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:54.304646 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:54.304707 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:54.304857 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:54.304857 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:54.304859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:54.304909 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.304910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.305913 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.305913 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.307157 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.307160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.307961 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.307963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.308064 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.308080 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 19:07:54.308135 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 19:07:54.308186 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":3,"thread_start_us":150,"threads_started":1}
I20260620 19:07:54.308319 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.304314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:44943 (local address 127.0.203.1:46529)
0620 19:07:54.304380 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:54.304395 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:54.304455 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:54.304661 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:54.304663 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:54.304665 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:54.304741 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.304743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.305791 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.305792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.307157 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.307160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.307962 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.307963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.308064 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.308068 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:07:54.308135 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:07:54.308187 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":70}
I20260620 19:07:54.308753 2224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeed0cd09f1c4f9eb2884642e4cc6f92" candidate_uuid: "a57b03d0719b40d5b05ce328f16736a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47b3e7139db49d48c3f64034a496f05" is_pre_election: true
I20260620 19:07:54.308769 2364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeed0cd09f1c4f9eb2884642e4cc6f92" candidate_uuid: "a57b03d0719b40d5b05ce328f16736a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78739bdb24214422921dc8329589abf7" is_pre_election: true
I20260620 19:07:54.308929 2364 raft_consensus.cc:2454] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a57b03d0719b40d5b05ce328f16736a8 in term 0.
I20260620 19:07:54.308929 2224 raft_consensus.cc:2454] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a57b03d0719b40d5b05ce328f16736a8 in term 0.
I20260620 19:07:54.309175 2018 leader_election.cc:302] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a57b03d0719b40d5b05ce328f16736a8, e47b3e7139db49d48c3f64034a496f05; no voters:
I20260620 19:07:54.309314 2440 raft_consensus.cc:2781] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:07:54.309373 2440 raft_consensus.cc:482] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:07:54.309397 2440 raft_consensus.cc:3037] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:54.310344 2440 raft_consensus.cc:504] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } }
I20260620 19:07:54.310492 2440 leader_election.cc:288] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [CANDIDATE]: Term 1 election: Requested vote from peers 78739bdb24214422921dc8329589abf7 (127.0.203.3:44943), e47b3e7139db49d48c3f64034a496f05 (127.0.203.2:40729)
I20260620 19:07:54.310658 2364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeed0cd09f1c4f9eb2884642e4cc6f92" candidate_uuid: "a57b03d0719b40d5b05ce328f16736a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78739bdb24214422921dc8329589abf7"
I20260620 19:07:54.310683 2224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeed0cd09f1c4f9eb2884642e4cc6f92" candidate_uuid: "a57b03d0719b40d5b05ce328f16736a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47b3e7139db49d48c3f64034a496f05"
I20260620 19:07:54.310745 2364 raft_consensus.cc:3037] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:54.310770 2224 raft_consensus.cc:3037] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:54.311676 2224 raft_consensus.cc:2454] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a57b03d0719b40d5b05ce328f16736a8 in term 1.
I20260620 19:07:54.311673 2364 raft_consensus.cc:2454] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a57b03d0719b40d5b05ce328f16736a8 in term 1.
I20260620 19:07:54.311831 2018 leader_election.cc:302] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a57b03d0719b40d5b05ce328f16736a8, e47b3e7139db49d48c3f64034a496f05; no voters:
I20260620 19:07:54.311926 2440 raft_consensus.cc:2781] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:07:54.312081 2440 raft_consensus.cc:686] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [term 1 LEADER]: Becoming Leader. State: Replica: a57b03d0719b40d5b05ce328f16736a8, State: Running, Role: LEADER
I20260620 19:07:54.312158 2440 consensus_queue.cc:237] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } }
I20260620 19:07:54.313133 1927 catalog_manager.cc:5495] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a57b03d0719b40d5b05ce328f16736a8 (127.0.203.1). New cstate: current_term: 1 leader_uuid: "a57b03d0719b40d5b05ce328f16736a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } health_report { overall_health: HEALTHY } } }
I20260620 19:07:54.332535 2432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.329832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51044 (local address 127.0.203.1:39453)
0620 19:07:54.329894 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.329896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.329907 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:07:54.329969 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.329969 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.329988 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.330040 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:07:54.330205 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.330693 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.331545 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.331747 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.331807 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.332032 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:07:54.332133 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:07:54.332198 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.332258 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.332281 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:07:54.334450 2224 raft_consensus.cc:1261] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05 [term 1 FOLLOWER]: Refusing update from remote peer a57b03d0719b40d5b05ce328f16736a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:54.334497 2364 raft_consensus.cc:1261] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7 [term 1 FOLLOWER]: Refusing update from remote peer a57b03d0719b40d5b05ce328f16736a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:54.334833 2440 consensus_queue.cc:1035] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:54.334969 2443 consensus_queue.cc:1035] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:54.341346 1927 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:07:54.341456 1927 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:07:54.342334 1927 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e: Sending DeleteTablet for 3 replicas of tablet eeed0cd09f1c4f9eb2884642e4cc6f92
I20260620 19:07:54.342707 2344 tablet_service.cc:1514] Processing DeleteTablet for tablet eeed0cd09f1c4f9eb2884642e4cc6f92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:54 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:47870
I20260620 19:07:54.342712 2064 tablet_service.cc:1514] Processing DeleteTablet for tablet eeed0cd09f1c4f9eb2884642e4cc6f92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:54 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:51042
I20260620 19:07:54.342768 2204 tablet_service.cc:1514] Processing DeleteTablet for tablet eeed0cd09f1c4f9eb2884642e4cc6f92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:54 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:39572
I20260620 19:07:54.342973 2455 tablet_replica.cc:330] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7: stopping tablet replica
I20260620 19:07:54.343096 2455 raft_consensus.cc:2229] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:07:54.343269 1927 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:07:54.343437 1927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:07:54.343744 2455 raft_consensus.cc:2258] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:54.343928 2457 tablet_replica.cc:330] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05: stopping tablet replica
I20260620 19:07:54.344050 2457 raft_consensus.cc:2229] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:07:54.344161 2457 raft_consensus.cc:2258] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:54.344322 2456 tablet_replica.cc:330] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8: stopping tablet replica
I20260620 19:07:54.344421 2456 raft_consensus.cc:2229] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:07:54.344525 2456 raft_consensus.cc:2258] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:54.344844 2455 ts_tablet_manager.cc:1905] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:54.344852 2064 tablet_service.cc:1467] Processing CreateTablet for tablet 6fcb87cc5c004244ae7c979b53076321 (DEFAULT_TABLE table=test-table [id=f47c8aa9a0b84b79a93b39c1785d8508]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:54.344908 2456 ts_tablet_manager.cc:1905] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:54.344969 2064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fcb87cc5c004244ae7c979b53076321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:54.346410 2434 tablet_bootstrap.cc:492] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8: Bootstrap starting.
I20260620 19:07:54.346766 2344 tablet_service.cc:1467] Processing CreateTablet for tablet 6fcb87cc5c004244ae7c979b53076321 (DEFAULT_TABLE table=test-table [id=f47c8aa9a0b84b79a93b39c1785d8508]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:54.346851 2434 tablet_bootstrap.cc:654] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:54.346868 2344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fcb87cc5c004244ae7c979b53076321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:54.347240 2455 ts_tablet_manager.cc:1918] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:07:54.347308 2455 log.cc:1196] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/eeed0cd09f1c4f9eb2884642e4cc6f92
I20260620 19:07:54.347291 2204 tablet_service.cc:1467] Processing CreateTablet for tablet 6fcb87cc5c004244ae7c979b53076321 (DEFAULT_TABLE table=test-table [id=f47c8aa9a0b84b79a93b39c1785d8508]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:54.347405 2204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fcb87cc5c004244ae7c979b53076321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:54.347445 2456 ts_tablet_manager.cc:1918] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:07:54.347519 2456 log.cc:1196] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-0/wal/wals/eeed0cd09f1c4f9eb2884642e4cc6f92
I20260620 19:07:54.347569 2455 ts_tablet_manager.cc:1939] T eeed0cd09f1c4f9eb2884642e4cc6f92 P 78739bdb24214422921dc8329589abf7: Deleting consensus metadata
I20260620 19:07:54.347769 2456 ts_tablet_manager.cc:1939] T eeed0cd09f1c4f9eb2884642e4cc6f92 P a57b03d0719b40d5b05ce328f16736a8: Deleting consensus metadata
I20260620 19:07:54.347967 1914 catalog_manager.cc:4841] TS 78739bdb24214422921dc8329589abf7 (127.0.203.3:44943): tablet eeed0cd09f1c4f9eb2884642e4cc6f92 (table test-table [id=af3b2e0261364840a0237438452b44b7]) successfully deleted
I20260620 19:07:54.348148 1913 catalog_manager.cc:4841] TS a57b03d0719b40d5b05ce328f16736a8 (127.0.203.1:39453): tablet eeed0cd09f1c4f9eb2884642e4cc6f92 (table test-table [id=af3b2e0261364840a0237438452b44b7]) successfully deleted
I20260620 19:07:54.348613 2434 tablet_bootstrap.cc:492] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8: No bootstrap required, opened a new log
I20260620 19:07:54.348655 2434 ts_tablet_manager.cc:1397] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:07:54.348726 2435 tablet_bootstrap.cc:492] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05: Bootstrap starting.
I20260620 19:07:54.348799 2434 raft_consensus.cc:348] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } }
I20260620 19:07:54.348888 2434 raft_consensus.cc:374] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:54.348914 2434 raft_consensus.cc:729] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57b03d0719b40d5b05ce328f16736a8, State: Initialized, Role: FOLLOWER
I20260620 19:07:54.348973 2434 consensus_queue.cc:260] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } }
I20260620 19:07:54.349066 2434 ts_tablet_manager.cc:1428] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:54.349145 2435 tablet_bootstrap.cc:654] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:54.349756 2435 tablet_bootstrap.cc:492] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05: No bootstrap required, opened a new log
I20260620 19:07:54.349807 2435 ts_tablet_manager.cc:1397] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:07:54.349948 2435 raft_consensus.cc:348] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } }
I20260620 19:07:54.350021 2435 raft_consensus.cc:374] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:54.350044 2435 raft_consensus.cc:729] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47b3e7139db49d48c3f64034a496f05, State: Initialized, Role: FOLLOWER
I20260620 19:07:54.350091 2435 consensus_queue.cc:260] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } }
I20260620 19:07:54.350179 2435 ts_tablet_manager.cc:1428] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:54.351244 2433 tablet_bootstrap.cc:492] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7: Bootstrap starting.
I20260620 19:07:54.351585 2433 tablet_bootstrap.cc:654] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:54.352226 2433 tablet_bootstrap.cc:492] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7: No bootstrap required, opened a new log
I20260620 19:07:54.352280 2433 ts_tablet_manager.cc:1397] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:07:54.352339 2457 ts_tablet_manager.cc:1905] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:54.352427 2433 raft_consensus.cc:348] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } }
I20260620 19:07:54.352545 2433 raft_consensus.cc:374] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:54.352569 2433 raft_consensus.cc:729] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78739bdb24214422921dc8329589abf7, State: Initialized, Role: FOLLOWER
I20260620 19:07:54.352610 2433 consensus_queue.cc:260] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } }
I20260620 19:07:54.352698 2433 ts_tablet_manager.cc:1428] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:54.354027 2457 ts_tablet_manager.cc:1918] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:07:54.354111 2457 log.cc:1196] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/eeed0cd09f1c4f9eb2884642e4cc6f92
I20260620 19:07:54.354382 2457 ts_tablet_manager.cc:1939] T eeed0cd09f1c4f9eb2884642e4cc6f92 P e47b3e7139db49d48c3f64034a496f05: Deleting consensus metadata
I20260620 19:07:54.354839 1914 catalog_manager.cc:4841] TS e47b3e7139db49d48c3f64034a496f05 (127.0.203.2:40729): tablet eeed0cd09f1c4f9eb2884642e4cc6f92 (table test-table [id=af3b2e0261364840a0237438452b44b7]) successfully deleted
I20260620 19:07:54.382324 2441 raft_consensus.cc:482] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:07:54.382437 2441 raft_consensus.cc:504] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } }
I20260620 19:07:54.382797 2441 leader_election.cc:288] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a57b03d0719b40d5b05ce328f16736a8 (127.0.203.1:39453), 78739bdb24214422921dc8329589abf7 (127.0.203.3:44943)
I20260620 19:07:54.386086 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.383205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:46119 (local address 127.0.203.3:44943)
0620 19:07:54.383250 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.383252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.383259 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:07:54.383282 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.383282 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.383291 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.383346 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:54.383557 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.384270 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.385311 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.385715 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.385786 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.385796 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.385972 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.386009 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:07:54.386538 2461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.382942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:44943 (local address 127.0.203.2:46119)
0620 19:07:54.383108 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:54.383145 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:54.383210 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:54.383368 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:54.383369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:54.383371 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:54.383451 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.383453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.384376 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.384377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.385219 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.385221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.385801 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.385803 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.385877 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.385882 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:07:54.385961 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 19:07:54.386001 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":69,"thread_start_us":58,"threads_started":1}
I20260620 19:07:54.386617 2432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.383152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:37275 (local address 127.0.203.1:39453)
0620 19:07:54.383194 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.383196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.383204 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:07:54.383232 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.383233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.383243 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.383295 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:54.383493 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.384260 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.385652 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.386168 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.386236 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.386245 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.386508 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.386541 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:07:54.386773 2364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fcb87cc5c004244ae7c979b53076321" candidate_uuid: "e47b3e7139db49d48c3f64034a496f05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78739bdb24214422921dc8329589abf7" is_pre_election: true
I20260620 19:07:54.386873 2364 raft_consensus.cc:2454] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47b3e7139db49d48c3f64034a496f05 in term 0.
I20260620 19:07:54.386943 2163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.382942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:39453 (local address 127.0.203.2:37275)
0620 19:07:54.382998 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:54.383010 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:54.383050 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:54.383344 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:54.383345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:54.383346 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:54.383421 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.383422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.384360 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.384361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.385553 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.385556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.386372 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.386373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.386448 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.386451 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:07:54.386504 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 19:07:54.386548 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":24}
I20260620 19:07:54.387151 2158 leader_election.cc:302] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78739bdb24214422921dc8329589abf7, e47b3e7139db49d48c3f64034a496f05; no voters:
I20260620 19:07:54.387286 2441 raft_consensus.cc:2781] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:07:54.387337 2441 raft_consensus.cc:482] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:07:54.387362 2441 raft_consensus.cc:3037] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:54.387509 2084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fcb87cc5c004244ae7c979b53076321" candidate_uuid: "e47b3e7139db49d48c3f64034a496f05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57b03d0719b40d5b05ce328f16736a8" is_pre_election: true
I20260620 19:07:54.387652 2084 raft_consensus.cc:2454] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47b3e7139db49d48c3f64034a496f05 in term 0.
I20260620 19:07:54.388248 2441 raft_consensus.cc:504] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } }
I20260620 19:07:54.388394 2441 leader_election.cc:288] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [CANDIDATE]: Term 1 election: Requested vote from peers a57b03d0719b40d5b05ce328f16736a8 (127.0.203.1:39453), 78739bdb24214422921dc8329589abf7 (127.0.203.3:44943)
I20260620 19:07:54.388525 2084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fcb87cc5c004244ae7c979b53076321" candidate_uuid: "e47b3e7139db49d48c3f64034a496f05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57b03d0719b40d5b05ce328f16736a8"
I20260620 19:07:54.388595 2084 raft_consensus.cc:3037] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:54.388597 2364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fcb87cc5c004244ae7c979b53076321" candidate_uuid: "e47b3e7139db49d48c3f64034a496f05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78739bdb24214422921dc8329589abf7"
I20260620 19:07:54.388659 2364 raft_consensus.cc:3037] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:54.389370 2364 raft_consensus.cc:2454] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47b3e7139db49d48c3f64034a496f05 in term 1.
I20260620 19:07:54.389370 2084 raft_consensus.cc:2454] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47b3e7139db49d48c3f64034a496f05 in term 1.
I20260620 19:07:54.389536 2158 leader_election.cc:302] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78739bdb24214422921dc8329589abf7, e47b3e7139db49d48c3f64034a496f05; no voters:
I20260620 19:07:54.389654 2441 raft_consensus.cc:2781] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:07:54.389827 2441 raft_consensus.cc:686] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [term 1 LEADER]: Becoming Leader. State: Replica: e47b3e7139db49d48c3f64034a496f05, State: Running, Role: LEADER
I20260620 19:07:54.389906 2441 consensus_queue.cc:237] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } }
I20260620 19:07:54.390609 1926 catalog_manager.cc:5495] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47b3e7139db49d48c3f64034a496f05 (127.0.203.2). New cstate: current_term: 1 leader_uuid: "e47b3e7139db49d48c3f64034a496f05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } health_report { overall_health: UNKNOWN } } }
W20260620 19:07:54.392925 2131 tablet.cc:2369] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 19:07:54 dist-test-slave-wd30 krb5kdc[1882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:07:54.428009 2469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.419919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42077 (local address 127.0.0.1:46466)
0620 19:07:54.420120 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:54.420129 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:54.420190 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:54.420403 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:54.420404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:54.420731 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:54.420782 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.420785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.421745 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.421746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.422269 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.422270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.422742 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.422743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.422793 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.423448 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:54.423452 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:54.424845 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:54.426835 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:54.426843 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:54.426844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:54.427058 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:54.427340 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:54.427341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:54.427341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:54.427401 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:54.427696 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:54.427702 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:54.427772 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 19:07:54.427840 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 19:07:54.427932 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":85,"threads_started":1}
I20260620 19:07:54.428122 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.420037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.0.203.62:42077)
0620 19:07:54.420116 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.420118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.420198 (+ 80us) server_negotiation.cc:407] Connection header received
0620 19:07:54.420264 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.420265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.420280 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.420351 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:54.420884 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.421637 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.422361 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.422649 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.422737 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.424956 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:54.424958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:54.424958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:54.424966 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:54.426752 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:54.427171 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:54.427172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:54.427173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:54.427257 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:54.427467 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:54.427468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:54.427468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:54.427623 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:54.427709 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.427898 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.428003 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:07:54.430656 1926 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:46466:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 19:07:54.430893 1926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:07:54.432439 2064 tablet_service.cc:1467] Processing CreateTablet for tablet 820bbec91b3e4d71965c215e0ef26907 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e8140b758d2e453690e8a21887d36af2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:07:54.432467 2204 tablet_service.cc:1467] Processing CreateTablet for tablet 820bbec91b3e4d71965c215e0ef26907 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e8140b758d2e453690e8a21887d36af2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:07:54.432614 2064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 820bbec91b3e4d71965c215e0ef26907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:54.432622 2204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 820bbec91b3e4d71965c215e0ef26907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:54.434229 2435 tablet_bootstrap.cc:492] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05: Bootstrap starting.
I20260620 19:07:54.434294 2344 tablet_service.cc:1467] Processing CreateTablet for tablet 820bbec91b3e4d71965c215e0ef26907 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e8140b758d2e453690e8a21887d36af2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:07:54.434432 2344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 820bbec91b3e4d71965c215e0ef26907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:54.434747 2435 tablet_bootstrap.cc:654] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:54.435206 2434 tablet_bootstrap.cc:492] T 820bbec91b3e4d71965c215e0ef26907 P a57b03d0719b40d5b05ce328f16736a8: Bootstrap starting.
I20260620 19:07:54.435681 2434 tablet_bootstrap.cc:654] T 820bbec91b3e4d71965c215e0ef26907 P a57b03d0719b40d5b05ce328f16736a8: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:54.435719 2433 tablet_bootstrap.cc:492] T 820bbec91b3e4d71965c215e0ef26907 P 78739bdb24214422921dc8329589abf7: Bootstrap starting.
I20260620 19:07:54.436604 2433 tablet_bootstrap.cc:654] T 820bbec91b3e4d71965c215e0ef26907 P 78739bdb24214422921dc8329589abf7: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:54.436673 2435 tablet_bootstrap.cc:492] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05: No bootstrap required, opened a new log
I20260620 19:07:54.436726 2435 ts_tablet_manager.cc:1397] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 19:07:54.436900 2435 raft_consensus.cc:348] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } }
I20260620 19:07:54.437000 2435 raft_consensus.cc:374] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:54.437024 2435 raft_consensus.cc:729] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47b3e7139db49d48c3f64034a496f05, State: Initialized, Role: FOLLOWER
I20260620 19:07:54.437078 2435 consensus_queue.cc:260] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } }
I20260620 19:07:54.437188 2435 ts_tablet_manager.cc:1428] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:54.437222 2433 tablet_bootstrap.cc:492] T 820bbec91b3e4d71965c215e0ef26907 P 78739bdb24214422921dc8329589abf7: No bootstrap required, opened a new log
I20260620 19:07:54.437264 2433 ts_tablet_manager.cc:1397] T 820bbec91b3e4d71965c215e0ef26907 P 78739bdb24214422921dc8329589abf7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 19:07:54.437254 2441 tablet_replica.cc:439] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } } }
I20260620 19:07:54.437347 2441 tablet_replica.cc:442] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:07:54.437417 2433 raft_consensus.cc:348] T 820bbec91b3e4d71965c215e0ef26907 P 78739bdb24214422921dc8329589abf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } }
I20260620 19:07:54.437481 2433 raft_consensus.cc:374] T 820bbec91b3e4d71965c215e0ef26907 P 78739bdb24214422921dc8329589abf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:54.437503 2433 raft_consensus.cc:729] T 820bbec91b3e4d71965c215e0ef26907 P 78739bdb24214422921dc8329589abf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78739bdb24214422921dc8329589abf7, State: Initialized, Role: FOLLOWER
I20260620 19:07:54.437544 2433 consensus_queue.cc:260] T 820bbec91b3e4d71965c215e0ef26907 P 78739bdb24214422921dc8329589abf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } }
I20260620 19:07:54.437624 2433 ts_tablet_manager.cc:1428] T 820bbec91b3e4d71965c215e0ef26907 P 78739bdb24214422921dc8329589abf7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:54.437731 2434 tablet_bootstrap.cc:492] T 820bbec91b3e4d71965c215e0ef26907 P a57b03d0719b40d5b05ce328f16736a8: No bootstrap required, opened a new log
I20260620 19:07:54.437783 2434 ts_tablet_manager.cc:1397] T 820bbec91b3e4d71965c215e0ef26907 P a57b03d0719b40d5b05ce328f16736a8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 19:07:54.437776 2439 tablet_replica.cc:439] T 820bbec91b3e4d71965c215e0ef26907 P 78739bdb24214422921dc8329589abf7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } } }
I20260620 19:07:54.437866 2439 tablet_replica.cc:442] T 820bbec91b3e4d71965c215e0ef26907 P 78739bdb24214422921dc8329589abf7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:07:54.438122 2434 raft_consensus.cc:348] T 820bbec91b3e4d71965c215e0ef26907 P a57b03d0719b40d5b05ce328f16736a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } }
I20260620 19:07:54.438210 2434 raft_consensus.cc:374] T 820bbec91b3e4d71965c215e0ef26907 P a57b03d0719b40d5b05ce328f16736a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:54.438234 2434 raft_consensus.cc:729] T 820bbec91b3e4d71965c215e0ef26907 P a57b03d0719b40d5b05ce328f16736a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57b03d0719b40d5b05ce328f16736a8, State: Initialized, Role: FOLLOWER
I20260620 19:07:54.438294 2434 consensus_queue.cc:260] T 820bbec91b3e4d71965c215e0ef26907 P a57b03d0719b40d5b05ce328f16736a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } }
I20260620 19:07:54.438405 2434 ts_tablet_manager.cc:1428] T 820bbec91b3e4d71965c215e0ef26907 P a57b03d0719b40d5b05ce328f16736a8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:07:54.438577 2443 tablet_replica.cc:439] T 820bbec91b3e4d71965c215e0ef26907 P a57b03d0719b40d5b05ce328f16736a8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } } }
I20260620 19:07:54.438660 2443 tablet_replica.cc:442] T 820bbec91b3e4d71965c215e0ef26907 P a57b03d0719b40d5b05ce328f16736a8: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 19:07:54.452278 2411 tablet.cc:2369] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:07:54.625828 2441 raft_consensus.cc:482] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:07:54.625942 2441 raft_consensus.cc:504] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } }
I20260620 19:07:54.626298 2441 leader_election.cc:288] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a57b03d0719b40d5b05ce328f16736a8 (127.0.203.1:39453), 78739bdb24214422921dc8329589abf7 (127.0.203.3:44943)
I20260620 19:07:54.626600 2084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820bbec91b3e4d71965c215e0ef26907" candidate_uuid: "e47b3e7139db49d48c3f64034a496f05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57b03d0719b40d5b05ce328f16736a8" is_pre_election: true
I20260620 19:07:54.626600 2364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820bbec91b3e4d71965c215e0ef26907" candidate_uuid: "e47b3e7139db49d48c3f64034a496f05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78739bdb24214422921dc8329589abf7" is_pre_election: true
I20260620 19:07:54.626716 2084 raft_consensus.cc:2454] T 820bbec91b3e4d71965c215e0ef26907 P a57b03d0719b40d5b05ce328f16736a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47b3e7139db49d48c3f64034a496f05 in term 0.
I20260620 19:07:54.626729 2364 raft_consensus.cc:2454] T 820bbec91b3e4d71965c215e0ef26907 P 78739bdb24214422921dc8329589abf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47b3e7139db49d48c3f64034a496f05 in term 0.
I20260620 19:07:54.626871 2158 leader_election.cc:302] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78739bdb24214422921dc8329589abf7, e47b3e7139db49d48c3f64034a496f05; no voters:
I20260620 19:07:54.626993 2441 raft_consensus.cc:2781] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:07:54.627046 2441 raft_consensus.cc:482] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:07:54.627079 2441 raft_consensus.cc:3037] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:54.627856 2441 raft_consensus.cc:504] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } }
I20260620 19:07:54.628021 2441 leader_election.cc:288] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05 [CANDIDATE]: Term 1 election: Requested vote from peers a57b03d0719b40d5b05ce328f16736a8 (127.0.203.1:39453), 78739bdb24214422921dc8329589abf7 (127.0.203.3:44943)
I20260620 19:07:54.628253 2364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820bbec91b3e4d71965c215e0ef26907" candidate_uuid: "e47b3e7139db49d48c3f64034a496f05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78739bdb24214422921dc8329589abf7"
I20260620 19:07:54.628261 2084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820bbec91b3e4d71965c215e0ef26907" candidate_uuid: "e47b3e7139db49d48c3f64034a496f05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57b03d0719b40d5b05ce328f16736a8"
I20260620 19:07:54.628355 2364 raft_consensus.cc:3037] T 820bbec91b3e4d71965c215e0ef26907 P 78739bdb24214422921dc8329589abf7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:54.628372 2084 raft_consensus.cc:3037] T 820bbec91b3e4d71965c215e0ef26907 P a57b03d0719b40d5b05ce328f16736a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:54.629060 2364 raft_consensus.cc:2454] T 820bbec91b3e4d71965c215e0ef26907 P 78739bdb24214422921dc8329589abf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47b3e7139db49d48c3f64034a496f05 in term 1.
I20260620 19:07:54.629099 2084 raft_consensus.cc:2454] T 820bbec91b3e4d71965c215e0ef26907 P a57b03d0719b40d5b05ce328f16736a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47b3e7139db49d48c3f64034a496f05 in term 1.
I20260620 19:07:54.629298 2157 leader_election.cc:302] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a57b03d0719b40d5b05ce328f16736a8, e47b3e7139db49d48c3f64034a496f05; no voters:
I20260620 19:07:54.629446 2441 raft_consensus.cc:2781] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:07:54.629520 2441 raft_consensus.cc:686] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05 [term 1 LEADER]: Becoming Leader. State: Replica: e47b3e7139db49d48c3f64034a496f05, State: Running, Role: LEADER
I20260620 19:07:54.629571 2441 consensus_queue.cc:237] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } }
I20260620 19:07:54.629853 2462 tablet_replica.cc:439] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05: TxnStatusTablet state changed. Reason: New leader e47b3e7139db49d48c3f64034a496f05. Latest consensus state: current_term: 1 leader_uuid: "e47b3e7139db49d48c3f64034a496f05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } } }
I20260620 19:07:54.629937 2462 tablet_replica.cc:442] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:07:54.630106 2474 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 19:07:54.630228 2474 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 19:07:54.630195 1925 catalog_manager.cc:5495] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47b3e7139db49d48c3f64034a496f05 (127.0.203.2). New cstate: current_term: 1 leader_uuid: "e47b3e7139db49d48c3f64034a496f05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } health_report { overall_health: UNKNOWN } } }
I20260620 19:07:54.668336 2431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.665546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39578 (local address 127.0.203.2:40729)
0620 19:07:54.665599 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.665600 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.665608 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:07:54.665703 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.665704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.665721 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.665777 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:07:54.665919 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.666402 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.667562 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.667747 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.667805 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.668007 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:07:54.668105 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:07:54.668161 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.668236 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.668254 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:07:54.668363 2469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.665468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:40729 (local address 127.0.0.1:39578)
0620 19:07:54.665553 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:54.665566 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:54.665619 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:54.665800 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:54.665801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:54.665802 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:07:54.665856 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.665858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.666473 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.666474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.667461 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.667465 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.667824 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.667825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.667893 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.667911 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:07:54.668169 (+ 258us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:07:54.668176 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 19:07:54.668231 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 19:07:54.668280 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 19:07:54.669641 2364 raft_consensus.cc:1261] T 820bbec91b3e4d71965c215e0ef26907 P 78739bdb24214422921dc8329589abf7 [term 1 FOLLOWER]: Refusing update from remote peer e47b3e7139db49d48c3f64034a496f05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:54.669729 2084 raft_consensus.cc:1261] T 820bbec91b3e4d71965c215e0ef26907 P a57b03d0719b40d5b05ce328f16736a8 [term 1 FOLLOWER]: Refusing update from remote peer e47b3e7139db49d48c3f64034a496f05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:54.669911 2462 consensus_queue.cc:1035] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:54.670068 2441 consensus_queue.cc:1035] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:54.670680 2439 tablet_replica.cc:439] T 820bbec91b3e4d71965c215e0ef26907 P 78739bdb24214422921dc8329589abf7: TxnStatusTablet state changed. Reason: New leader e47b3e7139db49d48c3f64034a496f05. Latest consensus state: current_term: 1 leader_uuid: "e47b3e7139db49d48c3f64034a496f05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } } }
I20260620 19:07:54.670809 2439 tablet_replica.cc:442] T 820bbec91b3e4d71965c215e0ef26907 P 78739bdb24214422921dc8329589abf7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:07:54.670747 2443 tablet_replica.cc:439] T 820bbec91b3e4d71965c215e0ef26907 P a57b03d0719b40d5b05ce328f16736a8: TxnStatusTablet state changed. Reason: New leader e47b3e7139db49d48c3f64034a496f05. Latest consensus state: current_term: 1 leader_uuid: "e47b3e7139db49d48c3f64034a496f05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } } }
I20260620 19:07:54.670856 2443 tablet_replica.cc:442] T 820bbec91b3e4d71965c215e0ef26907 P a57b03d0719b40d5b05ce328f16736a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:07:54.671413 2441 tablet_replica.cc:439] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e47b3e7139db49d48c3f64034a496f05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } } }
I20260620 19:07:54.671492 2441 tablet_replica.cc:442] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:07:54.671574 2439 tablet_replica.cc:439] T 820bbec91b3e4d71965c215e0ef26907 P 78739bdb24214422921dc8329589abf7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e47b3e7139db49d48c3f64034a496f05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } } }
I20260620 19:07:54.671631 2447 mvcc.cc:205] Tried to move back new op lower bound from 7299000216244736000 to 7299000216083066880. Current Snapshot: MvccSnapshot[applied={T|T < 7299000216244736000}]
I20260620 19:07:54.671665 2439 tablet_replica.cc:442] T 820bbec91b3e4d71965c215e0ef26907 P 78739bdb24214422921dc8329589abf7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:07:54.672009 2448 mvcc.cc:205] Tried to move back new op lower bound from 7299000216244736000 to 7299000216083066880. Current Snapshot: MvccSnapshot[applied={T|T < 7299000216244736000 or (T in {7299000216244736000})}]
I20260620 19:07:54.672435 2462 tablet_replica.cc:439] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e47b3e7139db49d48c3f64034a496f05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } } }
I20260620 19:07:54.672523 2462 tablet_replica.cc:442] T 820bbec91b3e4d71965c215e0ef26907 P e47b3e7139db49d48c3f64034a496f05: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:07:54.673177 2443 tablet_replica.cc:439] T 820bbec91b3e4d71965c215e0ef26907 P a57b03d0719b40d5b05ce328f16736a8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e47b3e7139db49d48c3f64034a496f05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 } } peers { permanent_uuid: "e47b3e7139db49d48c3f64034a496f05" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40729 } } peers { permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 } } }
I20260620 19:07:54.673264 2443 tablet_replica.cc:442] T 820bbec91b3e4d71965c215e0ef26907 P a57b03d0719b40d5b05ce328f16736a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:07:54.676182 2431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.673471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39592 (local address 127.0.203.2:40729)
0620 19:07:54.673513 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.673515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.673540 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:07:54.673602 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.673603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.673618 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.673672 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:07:54.673839 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.674316 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.675463 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.675662 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.675718 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.675867 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:07:54.675892 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:07:54.675974 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.676034 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.676051 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:07:54.681118 2431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.678662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:55985 (local address 127.0.203.2:40729)
0620 19:07:54.678716 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.678717 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.678842 (+ 125us) server_negotiation.cc:407] Connection header received
0620 19:07:54.678901 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.678902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.678910 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.678979 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:07:54.679132 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.679624 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.680440 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.680619 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.680696 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.680860 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:07:54.680894 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:07:54.680953 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.681010 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.681043 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":29}
I20260620 19:07:54.681149 2481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.678590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:40729 (local address 127.0.203.2:55985)
0620 19:07:54.678780 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:54.678789 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:54.678855 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:54.678980 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:54.678980 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:54.678982 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:07:54.679052 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.679053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.679725 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.679726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.680352 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:54.680353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.680715 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.680717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:54.680770 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.680787 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:07:54.680953 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:07:54.680955 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:07:54.681006 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:07:54.681061 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":3,"thread_start_us":104,"threads_started":1}
I20260620 19:07:54.683557 2084 raft_consensus.cc:1261] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8 [term 1 FOLLOWER]: Refusing update from remote peer e47b3e7139db49d48c3f64034a496f05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:54.683583 2364 raft_consensus.cc:1261] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7 [term 1 FOLLOWER]: Refusing update from remote peer e47b3e7139db49d48c3f64034a496f05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:54.683871 2482 consensus_queue.cc:1035] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57b03d0719b40d5b05ce328f16736a8" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:54.684000 2441 consensus_queue.cc:1035] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78739bdb24214422921dc8329589abf7" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:54.686161 2446 mvcc.cc:205] Tried to move back new op lower bound from 7299000216302329856 to 7299000215101493248. Current Snapshot: MvccSnapshot[applied={T|T < 7299000216302329856}]
I20260620 19:07:54.698652 1925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:07:54.698792 1925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:07:54.699568 1925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc0b0b475a914ac3b4f2b1bd5035435e: Sending DeleteTablet for 3 replicas of tablet 6fcb87cc5c004244ae7c979b53076321
I20260620 19:07:54.699879 2064 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fcb87cc5c004244ae7c979b53076321 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:54 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:51042
I20260620 19:07:54.699890 2344 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fcb87cc5c004244ae7c979b53076321 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:54 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:47870
I20260620 19:07:54.699994 2455 tablet_replica.cc:330] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7: stopping tablet replica
I20260620 19:07:54.700099 2456 tablet_replica.cc:330] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8: stopping tablet replica
I20260620 19:07:54.700171 2456 raft_consensus.cc:2229] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:07:54.700160 2455 raft_consensus.cc:2229] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:07:54.700213 2456 raft_consensus.cc:2258] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:54.700228 2455 raft_consensus.cc:2258] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:54.700372 2204 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fcb87cc5c004244ae7c979b53076321 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:54 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:39572
I20260620 19:07:54.700464 2457 tablet_replica.cc:330] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05: stopping tablet replica
I20260620 19:07:54.700516 2457 raft_consensus.cc:2229] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:07:54.700611 2457 raft_consensus.cc:2258] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:54.700695 2455 ts_tablet_manager.cc:1905] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:54.700943 2457 ts_tablet_manager.cc:1905] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:54.702420 2455 ts_tablet_manager.cc:1918] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:07:54.702445 2457 ts_tablet_manager.cc:1918] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:07:54.702502 2457 log.cc:1196] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/6fcb87cc5c004244ae7c979b53076321
I20260620 19:07:54.702504 2455 log.cc:1196] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/6fcb87cc5c004244ae7c979b53076321
I20260620 19:07:54.702737 2457 ts_tablet_manager.cc:1939] T 6fcb87cc5c004244ae7c979b53076321 P e47b3e7139db49d48c3f64034a496f05: Deleting consensus metadata
I20260620 19:07:54.702775 2455 ts_tablet_manager.cc:1939] T 6fcb87cc5c004244ae7c979b53076321 P 78739bdb24214422921dc8329589abf7: Deleting consensus metadata
I20260620 19:07:54.703171 1914 catalog_manager.cc:4841] TS 78739bdb24214422921dc8329589abf7 (127.0.203.3:44943): tablet 6fcb87cc5c004244ae7c979b53076321 (table test-table [id=f47c8aa9a0b84b79a93b39c1785d8508]) successfully deleted
I20260620 19:07:54.703265 1914 catalog_manager.cc:4841] TS e47b3e7139db49d48c3f64034a496f05 (127.0.203.2:40729): tablet 6fcb87cc5c004244ae7c979b53076321 (table test-table [id=f47c8aa9a0b84b79a93b39c1785d8508]) successfully deleted
I20260620 19:07:54.703629 2456 ts_tablet_manager.cc:1905] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:54.705329 2456 ts_tablet_manager.cc:1918] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:07:54.705487 2456 log.cc:1196] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982468536772-812-0/minicluster-data/ts-0/wal/wals/6fcb87cc5c004244ae7c979b53076321
I20260620 19:07:54.705888 2456 ts_tablet_manager.cc:1939] T 6fcb87cc5c004244ae7c979b53076321 P a57b03d0719b40d5b05ce328f16736a8: Deleting consensus metadata
I20260620 19:07:54.706301 1913 catalog_manager.cc:4841] TS a57b03d0719b40d5b05ce328f16736a8 (127.0.203.1:39453): tablet 6fcb87cc5c004244ae7c979b53076321 (table test-table [id=f47c8aa9a0b84b79a93b39c1785d8508]) successfully deleted
Jun 20 19:07:54 dist-test-slave-wd30 krb5kdc[1882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982474, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.203.1@KRBTEST.COM
I20260620 19:07:54.713856 2432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.703764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51058 (local address 127.0.203.1:39453)
0620 19:07:54.703815 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.703818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.704097 (+ 279us) server_negotiation.cc:407] Connection header received
0620 19:07:54.704130 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.704131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.704143 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.704214 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:54.704946 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.705722 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.706781 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.707096 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.707214 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.710492 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:54.710505 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:54.710505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:54.710514 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:54.712471 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:54.712906 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:54.712910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:54.712911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:54.712984 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:54.713185 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:54.713186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:54.713186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:54.713394 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:54.713566 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.713653 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.713751 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 19:07:54.714136 2124 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51058
I20260620 19:07:54.722270 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:54.715234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46482 (local address 127.0.203.62:42077)
0620 19:07:54.715307 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:54.715309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:54.715382 (+ 73us) server_negotiation.cc:407] Connection header received
0620 19:07:54.715462 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:54.715462 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:54.715471 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:54.715529 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:54.716221 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.716724 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.717758 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:54.717953 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:54.718018 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:54.719491 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:54.719493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:54.719493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:54.719502 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:54.721032 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:54.721475 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:54.721476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:54.721477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:54.721537 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:54.721762 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:54.721763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:54.721763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:54.721881 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:54.721957 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:54.722134 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 19:07:54.722203 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 19:07:54.722766 1925 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46482
I20260620 19:07:54.723547 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 1999
I20260620 19:07:54.730792 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 2139
I20260620 19:07:54.736683 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 2279
I20260620 19:07:54.742075 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 1895
2026-06-20T19:07:54Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2330 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:07:54 dist-test-slave-wd30 krb5kdc[2492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:07:54 dist-test-slave-wd30 krb5kdc[2492](info): set up 2 sockets
Jun 20 19:07:54 dist-test-slave-wd30 krb5kdc[2492](info): commencing operation
krb5kdc: starting...
W20260620 19:07:55.782763 812 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:07:55 dist-test-slave-wd30 krb5kdc[2492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:07:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:07:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:07:55.900175 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:40855
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:41433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:40855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:55.982182 2505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:55.982400 2505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:55.982436 2505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:55.983585 2505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:07:55.983641 2505 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 19:07:55.983665 2505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:55.983698 2505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:55.983721 2505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:07:55.983754 2505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:07:55.985117 2505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:41433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:40855
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:40855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:07:55.985375 2505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:55.985589 2505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:55.990357 2511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:55.990342 2513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:55.990330 2510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:55.990589 2505 server_base.cc:1034] running on GCE node
I20260620 19:07:55.990844 2505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:55.991113 2505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:55.992304 2505 hybrid_clock.cc:648] HybridClock initialized: now 1781982475992294 us; error 31 us; skew 500 ppm
Jun 20 19:07:55 dist-test-slave-wd30 krb5kdc[2492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:55.994742 2505 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:07:55.995237 2505 webserver.cc:460] Webserver started at http://127.0.203.62:36503/ using document root <none> and password file <none>
I20260620 19:07:55.995486 2505 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:55.995532 2505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:55.995676 2505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:55.996758 2505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "e467ed4e966a48b193c5df2a00b50f8b"
format_stamp: "Formatted at 2026-06-20 19:07:55 on dist-test-slave-wd30"
server_key: "6fbd3bdab3ad2b8a8f8c4fb45ed3d664"
server_key_iv: "b2c932827c7ef41730190844cda831fe"
server_key_version: "encryptionkey@0"
I20260620 19:07:55.997143 2505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "e467ed4e966a48b193c5df2a00b50f8b"
format_stamp: "Formatted at 2026-06-20 19:07:55 on dist-test-slave-wd30"
server_key: "6fbd3bdab3ad2b8a8f8c4fb45ed3d664"
server_key_iv: "b2c932827c7ef41730190844cda831fe"
server_key_version: "encryptionkey@0"
I20260620 19:07:55.998548 2505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:07:55.999454 2520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:55.999644 2505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 19:07:55.999738 2505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "e467ed4e966a48b193c5df2a00b50f8b"
format_stamp: "Formatted at 2026-06-20 19:07:55 on dist-test-slave-wd30"
server_key: "6fbd3bdab3ad2b8a8f8c4fb45ed3d664"
server_key_iv: "b2c932827c7ef41730190844cda831fe"
server_key_version: "encryptionkey@0"
I20260620 19:07:55.999841 2505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:56.007993 2505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:56.010094 2505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:56.010255 2505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:56.014966 2505 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:40855
I20260620 19:07:56.015040 2582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:40855 every 8 connection(s)
I20260620 19:07:56.015496 2505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:07:56.016208 2583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:56.018555 2583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b: Bootstrap starting.
I20260620 19:07:56.019208 2583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:56.019479 2583 log.cc:824] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:56.020174 2583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b: No bootstrap required, opened a new log
I20260620 19:07:56.022276 2583 raft_consensus.cc:348] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e467ed4e966a48b193c5df2a00b50f8b" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 40855 } }
I20260620 19:07:56.022431 2583 raft_consensus.cc:374] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:56.022454 2583 raft_consensus.cc:729] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e467ed4e966a48b193c5df2a00b50f8b, State: Initialized, Role: FOLLOWER
I20260620 19:07:56.022610 2583 consensus_queue.cc:260] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e467ed4e966a48b193c5df2a00b50f8b" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 40855 } }
I20260620 19:07:56.022701 2583 raft_consensus.cc:388] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:07:56.022727 2583 raft_consensus.cc:482] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:07:56.022763 2583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:56.023394 2583 raft_consensus.cc:504] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e467ed4e966a48b193c5df2a00b50f8b" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 40855 } }
I20260620 19:07:56.023487 2583 leader_election.cc:302] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e467ed4e966a48b193c5df2a00b50f8b; no voters:
I20260620 19:07:56.023618 2583 leader_election.cc:288] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:07:56.023689 2586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:07:56.023922 2586 raft_consensus.cc:686] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b [term 1 LEADER]: Becoming Leader. State: Replica: e467ed4e966a48b193c5df2a00b50f8b, State: Running, Role: LEADER
I20260620 19:07:56.023933 2583 sys_catalog.cc:564] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:07:56.024034 2586 consensus_queue.cc:237] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e467ed4e966a48b193c5df2a00b50f8b" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 40855 } }
I20260620 19:07:56.024495 2587 sys_catalog.cc:455] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e467ed4e966a48b193c5df2a00b50f8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e467ed4e966a48b193c5df2a00b50f8b" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 40855 } } }
I20260620 19:07:56.024528 2588 sys_catalog.cc:455] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e467ed4e966a48b193c5df2a00b50f8b. Latest consensus state: current_term: 1 leader_uuid: "e467ed4e966a48b193c5df2a00b50f8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e467ed4e966a48b193c5df2a00b50f8b" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 40855 } } }
I20260620 19:07:56.024698 2588 sys_catalog.cc:458] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b [sys.catalog]: This master's current role is: LEADER
I20260620 19:07:56.025043 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 2505
I20260620 19:07:56.025139 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:07:56.025161 2595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:07:56.025345 812 external_mini_cluster.cc:1428] Setting key 459711f0998701a0a5a6659e74f9fc4e
I20260620 19:07:56.025655 2587 sys_catalog.cc:458] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b [sys.catalog]: This master's current role is: LEADER
I20260620 19:07:56.025795 2595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:07:56.027534 2595 catalog_manager.cc:1347] Generated new cluster ID: 3892ba7c682b410f8f44e202c5967081
I20260620 19:07:56.027585 2595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:07:56 dist-test-slave-wd30 krb5kdc[2492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:07:56.035250 2595 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:07:56.035945 2595 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:07:56.038028 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.026020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35648 (local address 127.0.203.62:40855)
0620 19:07:56.026218 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:56.026228 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:56.026264 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:07:56.026453 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:56.026461 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:56.026523 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:56.026683 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:56.027529 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.028965 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.030037 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.030660 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.030763 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.033835 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:56.033840 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:56.033841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:56.033855 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:56.036094 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:56.036548 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:56.036551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:56.036551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:56.036670 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:56.036910 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:56.036911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:56.036911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:56.037215 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:56.037386 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:56.037505 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 19:07:56.037779 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260620 19:07:56.045555 2595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b: Generated new TSK 0
I20260620 19:07:56.045858 2595 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
I20260620 19:07:56.073524 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:40855
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:41433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:56.160612 2609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:56.160854 2609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:56.160898 2609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:56.162111 2609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:56.162178 2609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:56.162223 2609 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 19:07:56.162276 2609 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:07:56.162318 2609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:07:56.163908 2609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:41433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.203.62:40855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:07:56.164177 2609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:56.164420 2609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:07:56.169377 2609 server_base.cc:1034] running on GCE node
W20260620 19:07:56.169308 2617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:56.169351 2614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:56.169323 2615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:56.169826 2609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:56.170027 2609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:56.171211 2609 hybrid_clock.cc:648] HybridClock initialized: now 1781982476171192 us; error 30 us; skew 500 ppm
Jun 20 19:07:56 dist-test-slave-wd30 krb5kdc[2492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:56.173628 2609 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:07:56.173996 2609 webserver.cc:460] Webserver started at http://127.0.203.1:38113/ using document root <none> and password file <none>
I20260620 19:07:56.174182 2609 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:56.174222 2609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:56.174315 2609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:56.175352 2609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "03fee9c5de834f9695a7976cb0503ca3"
format_stamp: "Formatted at 2026-06-20 19:07:56 on dist-test-slave-wd30"
server_key: "3637d744a42c01aa3da24ad568b4f6fc"
server_key_iv: "95fe3bdf56232d986e14b8648f1de96a"
server_key_version: "encryptionkey@0"
I20260620 19:07:56.175650 2609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "03fee9c5de834f9695a7976cb0503ca3"
format_stamp: "Formatted at 2026-06-20 19:07:56 on dist-test-slave-wd30"
server_key: "3637d744a42c01aa3da24ad568b4f6fc"
server_key_iv: "95fe3bdf56232d986e14b8648f1de96a"
server_key_version: "encryptionkey@0"
I20260620 19:07:56.176855 2609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:07:56.177521 2624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:56.177716 2609 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:07:56.177815 2609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "03fee9c5de834f9695a7976cb0503ca3"
format_stamp: "Formatted at 2026-06-20 19:07:56 on dist-test-slave-wd30"
server_key: "3637d744a42c01aa3da24ad568b4f6fc"
server_key_iv: "95fe3bdf56232d986e14b8648f1de96a"
server_key_version: "encryptionkey@0"
I20260620 19:07:56.177897 2609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:56.189904 2609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:56.192088 2609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:56.192229 2609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:56.192695 2609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:56.192754 2609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:56.192801 2609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:56.192855 2609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:56.199779 2609 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:40833
I20260620 19:07:56.199839 2739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:40833 every 8 connection(s)
Jun 20 19:07:56 dist-test-slave-wd30 krb5kdc[2492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:07:56.200199 2609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:07:56.203831 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.193214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:52655 (local address 127.0.203.62:40855)
0620 19:07:56.193288 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:56.193291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:56.193473 (+ 182us) server_negotiation.cc:407] Connection header received
0620 19:07:56.193987 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:56.193988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:56.194004 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:56.194066 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:56.194944 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.195644 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.196739 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.197072 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.197147 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.200727 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:56.200728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:56.200729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:56.200737 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:56.202297 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:56.202750 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:56.202751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:56.202751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:56.202810 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:56.203108 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:56.203109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:56.203109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:56.203281 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:56.203339 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:56.203672 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 19:07:56.203762 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:07:56.204128 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.193201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:40855 (local address 127.0.203.1:52655)
0620 19:07:56.193367 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:56.193404 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:56.193884 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:56.194148 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:56.194156 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:56.194555 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:56.194852 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.194860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.195767 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.195768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.196632 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.196634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.197198 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.197200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.197388 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.198646 (+ 1258us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:56.198660 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:56.200592 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:56.202388 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:56.202395 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:56.202397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:56.202666 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:56.202905 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:56.202906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:56.202907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:56.203036 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:56.203369 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:56.203376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:56.203511 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 19:07:56.203656 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 19:07:56.203819 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":107,"threads_started":1}
I20260620 19:07:56.204702 2740 heartbeater.cc:347] Connected to a master server at 127.0.203.62:40855
I20260620 19:07:56.204811 2740 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:56.205003 2740 heartbeater.cc:511] Master 127.0.203.62:40855 requested a full tablet report, sending...
I20260620 19:07:56.205561 2537 ts_manager.cc:194] Registered new tserver with Master: 03fee9c5de834f9695a7976cb0503ca3 (127.0.203.1:40833)
I20260620 19:07:56.206283 2537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:52655
I20260620 19:07:56.208660 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 2609
I20260620 19:07:56.208745 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:07:56.208957 812 external_mini_cluster.cc:1428] Setting key 1c1dfd6e8e062b80178860ff429edcd6
I20260620 19:07:56.213065 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.205734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:40855 (local address 127.0.203.1:59819)
0620 19:07:56.205875 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:56.205889 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:56.205972 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:56.206102 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:56.206103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:56.206290 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:56.206344 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.206365 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.207079 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.207080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.207636 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.207638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.208036 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.208037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.208124 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.208593 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:56.208597 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:56.208980 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:56.211972 (+ 2992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:56.211973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:56.211974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:56.212155 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:56.212376 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:56.212377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:56.212377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:56.212433 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:56.212746 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:56.212747 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:56.212797 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 19:07:56.212891 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 19:07:56.212992 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260620 19:07:56.213169 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.205775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:59819 (local address 127.0.203.62:40855)
0620 19:07:56.205811 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:56.205813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:56.205947 (+ 134us) server_negotiation.cc:407] Connection header received
0620 19:07:56.206030 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:56.206031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:56.206041 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:56.206096 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:56.206478 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.206981 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.207747 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.207942 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.208034 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.209121 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:56.209122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:56.209122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:56.209130 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:56.211880 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:56.212238 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:56.212240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:56.212240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:56.212306 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:56.212497 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:56.212498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:56.212499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:56.212648 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:56.212717 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:56.212941 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 19:07:56.213033 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for kudu/127.0.203.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
I20260620 19:07:56.240875 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:40855
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:41433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:56.324522 2749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:56.324836 2749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:56.324865 2749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:56.327145 2749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:56.327212 2749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:56.327239 2749 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 19:07:56.327252 2749 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:07:56.327273 2749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:07:56.328857 2749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:41433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.203.62:40855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:07:56.329077 2749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:56.329267 2749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:56.334645 2757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:56.334658 2754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:56.334631 2755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:56.334965 2749 server_base.cc:1034] running on GCE node
I20260620 19:07:56.335245 2749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:56.335549 2749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:56.336720 2749 hybrid_clock.cc:648] HybridClock initialized: now 1781982476336711 us; error 34 us; skew 500 ppm
Jun 20 19:07:56 dist-test-slave-wd30 krb5kdc[2492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:56.339481 2749 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:07:56.340000 2749 webserver.cc:460] Webserver started at http://127.0.203.2:40571/ using document root <none> and password file <none>
I20260620 19:07:56.340260 2749 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:56.340318 2749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:56.340459 2749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:56.341589 2749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "f315f8c5ed9f4fe38f59de5c7f238516"
format_stamp: "Formatted at 2026-06-20 19:07:56 on dist-test-slave-wd30"
server_key: "131d00899a82b8b2481298128b7533a1"
server_key_iv: "1c06c1fcef67d4f9b6a1c86baf507454"
server_key_version: "encryptionkey@0"
I20260620 19:07:56.341969 2749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "f315f8c5ed9f4fe38f59de5c7f238516"
format_stamp: "Formatted at 2026-06-20 19:07:56 on dist-test-slave-wd30"
server_key: "131d00899a82b8b2481298128b7533a1"
server_key_iv: "1c06c1fcef67d4f9b6a1c86baf507454"
server_key_version: "encryptionkey@0"
I20260620 19:07:56.343438 2749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:07:56.344251 2764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:56.344434 2749 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:07:56.344543 2749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "f315f8c5ed9f4fe38f59de5c7f238516"
format_stamp: "Formatted at 2026-06-20 19:07:56 on dist-test-slave-wd30"
server_key: "131d00899a82b8b2481298128b7533a1"
server_key_iv: "1c06c1fcef67d4f9b6a1c86baf507454"
server_key_version: "encryptionkey@0"
I20260620 19:07:56.344621 2749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:56.355949 2749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:56.358057 2749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:56.358242 2749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:56.358826 2749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:56.358889 2749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:56.358971 2749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:56.359023 2749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:07:56 dist-test-slave-wd30 krb5kdc[2492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:07:56.366492 2749 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:45187
I20260620 19:07:56.366570 2879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:45187 every 8 connection(s)
I20260620 19:07:56.366915 2749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:07:56.369230 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.359259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:39853 (local address 127.0.203.62:40855)
0620 19:07:56.359327 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:56.359330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:56.359690 (+ 360us) server_negotiation.cc:407] Connection header received
0620 19:07:56.360438 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:56.360440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:56.360450 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:56.360530 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:56.361525 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.362368 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.363684 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.363898 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.364004 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.366277 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:56.366279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:56.366279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:56.366286 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:56.367861 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:56.368275 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:56.368276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:56.368276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:56.368339 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:56.368598 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:56.368599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:56.368599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:56.368717 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:56.368791 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:56.369065 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 19:07:56.369139 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 19:07:56.369417 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.359284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:40855 (local address 127.0.203.2:39853)
0620 19:07:56.359519 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:56.359573 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:56.360326 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:56.360628 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:56.360633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:56.361071 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:56.361404 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.361412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.362469 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.362470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.363591 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.363593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.364007 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.364008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.364143 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.364656 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:56.364664 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:56.366173 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:56.367959 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:56.367965 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:56.367967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:56.368185 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:56.368407 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:56.368407 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:56.368408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:56.368528 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:56.368796 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:56.368802 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:56.368937 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 19:07:56.369042 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 19:07:56.369195 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":135,"threads_started":1}
I20260620 19:07:56.369904 2880 heartbeater.cc:347] Connected to a master server at 127.0.203.62:40855
I20260620 19:07:56.370005 2880 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:56.370177 2880 heartbeater.cc:511] Master 127.0.203.62:40855 requested a full tablet report, sending...
I20260620 19:07:56.370543 2537 ts_manager.cc:194] Registered new tserver with Master: f315f8c5ed9f4fe38f59de5c7f238516 (127.0.203.2:45187)
I20260620 19:07:56.371054 2537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:39853
I20260620 19:07:56.375852 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.370933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:40761 (local address 127.0.203.62:40855)
0620 19:07:56.370975 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:56.370977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:56.371083 (+ 106us) server_negotiation.cc:407] Connection header received
0620 19:07:56.371199 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:56.371200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:56.371210 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:56.371246 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:56.371570 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.372122 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.372765 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.372933 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.372981 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.373837 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:56.373838 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:56.373838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:56.373844 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:56.374954 (+ 1110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:56.375296 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:56.375296 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:56.375297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:56.375339 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:56.375480 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:56.375481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:56.375481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:56.375598 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:56.375645 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:56.375747 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 19:07:56.375799 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:07:56.375859 2886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.370839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:40855 (local address 127.0.203.2:40761)
0620 19:07:56.371017 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:56.371027 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:56.371098 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:56.371303 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:56.371304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:56.371444 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:56.371499 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.371500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.372206 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.372206 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.372698 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.372699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.372993 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.372994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.373049 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.373466 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:56.373470 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:56.373733 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:56.375023 (+ 1290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:56.375024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:56.375025 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:56.375232 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:56.375394 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:56.375395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:56.375395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:56.375444 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:56.375648 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:56.375649 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:56.375681 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 19:07:56.375730 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:07:56.375801 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":95,"threads_started":1}
I20260620 19:07:56.376680 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 2749
I20260620 19:07:56.376797 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:07:56.376966 812 external_mini_cluster.cc:1428] Setting key 39372aa3b0a892986238b238a15f198b
WARNING: no policy specified for kudu/127.0.203.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
I20260620 19:07:56.402678 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:40855
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:41433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:07:56.486155 2889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:07:56.486351 2889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:07:56.486378 2889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:07:56.487609 2889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:07:56.487659 2889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:07:56.487684 2889 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 19:07:56.487735 2889 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:07:56.487769 2889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:07:56.489223 2889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:41433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.203.62:40855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:07:56.489478 2889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:07:56.489722 2889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:07:56.494831 2897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:07:56.494856 2895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:56.495030 2889 server_base.cc:1034] running on GCE node
W20260620 19:07:56.495088 2894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:07:56.495369 2889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:07:56.495633 2889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:07:56.496803 2889 hybrid_clock.cc:648] HybridClock initialized: now 1781982476496806 us; error 44 us; skew 500 ppm
Jun 20 19:07:56 dist-test-slave-wd30 krb5kdc[2492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:07:56.499778 2889 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:07:56.500383 2889 webserver.cc:460] Webserver started at http://127.0.203.3:39351/ using document root <none> and password file <none>
I20260620 19:07:56.500694 2889 fs_manager.cc:359] Metadata directory not provided
I20260620 19:07:56.500783 2889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:07:56.500941 2889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:07:56.502503 2889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "068176828f894b1599b47b45654b36c8"
format_stamp: "Formatted at 2026-06-20 19:07:56 on dist-test-slave-wd30"
server_key: "fd1c01212f338b33395ed3b41b654060"
server_key_iv: "beac1e994a7d4bb9fde7e573567bc01d"
server_key_version: "encryptionkey@0"
I20260620 19:07:56.502969 2889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "068176828f894b1599b47b45654b36c8"
format_stamp: "Formatted at 2026-06-20 19:07:56 on dist-test-slave-wd30"
server_key: "fd1c01212f338b33395ed3b41b654060"
server_key_iv: "beac1e994a7d4bb9fde7e573567bc01d"
server_key_version: "encryptionkey@0"
I20260620 19:07:56.505720 2889 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.001s
I20260620 19:07:56.506820 2904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:56.507038 2889 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:07:56.507162 2889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "068176828f894b1599b47b45654b36c8"
format_stamp: "Formatted at 2026-06-20 19:07:56 on dist-test-slave-wd30"
server_key: "fd1c01212f338b33395ed3b41b654060"
server_key_iv: "beac1e994a7d4bb9fde7e573567bc01d"
server_key_version: "encryptionkey@0"
I20260620 19:07:56.507256 2889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:07:56.538682 2889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:07:56.541271 2889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:07:56.541450 2889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:07:56.541985 2889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:07:56.542052 2889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:56.542117 2889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:07:56.542146 2889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:56.549767 2889 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:41499
Jun 20 19:07:56 dist-test-slave-wd30 krb5kdc[2492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:07:56.549988 3019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:41499 every 8 connection(s)
I20260620 19:07:56.550221 2889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:07:56.553043 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.542569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:59565 (local address 127.0.203.62:40855)
0620 19:07:56.542641 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:56.542643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:56.542878 (+ 235us) server_negotiation.cc:407] Connection header received
0620 19:07:56.543505 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:56.543506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:56.543521 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:56.543573 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:56.544520 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.545306 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.547201 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.547535 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.547617 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.550547 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:56.550550 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:56.550550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:56.550558 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:56.551789 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:56.552175 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:56.552176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:56.552176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:56.552229 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:56.552471 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:56.552471 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:56.552471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:56.552580 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:56.552631 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:56.552910 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 19:07:56.552974 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:07:56.553362 2913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.542554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:40855 (local address 127.0.203.3:59565)
0620 19:07:56.542748 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:56.542784 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:56.543374 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:56.543689 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:56.543695 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:56.544085 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:56.544425 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.544432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.545414 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.545415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.547046 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.547050 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.547654 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.547656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.547829 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.548494 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:56.548506 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:56.550441 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:56.551868 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:56.551874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:56.551876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:56.552097 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:56.552307 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:56.552308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:56.552308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:56.552409 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:56.552638 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:56.552642 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:56.552793 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 19:07:56.552879 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 19:07:56.553004 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":107,"threads_started":1}
I20260620 19:07:56.553908 3020 heartbeater.cc:347] Connected to a master server at 127.0.203.62:40855
I20260620 19:07:56.553982 3020 heartbeater.cc:464] Registering TS with master...
I20260620 19:07:56.554168 3020 heartbeater.cc:511] Master 127.0.203.62:40855 requested a full tablet report, sending...
I20260620 19:07:56.554596 2537 ts_manager.cc:194] Registered new tserver with Master: 068176828f894b1599b47b45654b36c8 (127.0.203.3:41499)
I20260620 19:07:56.555104 2537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:59565
I20260620 19:07:56.558513 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 2889
I20260620 19:07:56.558602 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:07:56.558813 812 external_mini_cluster.cc:1428] Setting key d7362b0b0519a1191374f99e314f6a4a
I20260620 19:07:56.559711 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:07:56.561014 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.555086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:43461 (local address 127.0.203.62:40855)
0620 19:07:56.555150 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:56.555153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:56.555229 (+ 76us) server_negotiation.cc:407] Connection header received
0620 19:07:56.555319 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:56.555320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:56.555331 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:56.555382 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:56.555684 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.556179 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.556905 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.557106 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.557190 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.558133 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:56.558135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:56.558135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:56.558142 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:56.559860 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:56.560174 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:56.560175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:56.560175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:56.560244 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:56.560467 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:56.560468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:56.560468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:56.560587 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:56.560662 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:56.560835 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 19:07:56.560917 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:07:56.561030 3026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.555012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:40855 (local address 127.0.203.3:43461)
0620 19:07:56.555166 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:56.555176 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:56.555253 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:56.555385 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:56.555386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:56.555553 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:56.555604 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.555606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.556283 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.556284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.556806 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.556808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.557190 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.557191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.557273 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.557690 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:56.557695 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:56.558007 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:56.559922 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:56.559923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:56.559924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:56.560100 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:56.560335 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:56.560336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:56.560336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:56.560402 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:56.560654 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:56.560655 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:56.560703 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 19:07:56.560806 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 19:07:56.560917 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
Jun 20 19:07:56 dist-test-slave-wd30 krb5kdc[2492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:07:56 dist-test-slave-wd30 krb5kdc[2492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:07:56.582021 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.575496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37868 (local address 127.0.203.62:40855)
0620 19:07:56.575559 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:56.575561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:56.575588 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:07:56.575608 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:56.575609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:56.575617 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:56.575656 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:56.576177 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.576654 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.577349 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.577520 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.577571 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.579611 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:56.579613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:56.579613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:56.579619 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:56.581125 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:56.581416 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:56.581416 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:56.581417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:56.581465 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:56.581657 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:56.581657 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:56.581657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:56.581747 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:56.581822 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:56.581902 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 19:07:56.581959 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:07:56.583496 2537 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:07:56.585057 2537 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:07:56.591159 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.587332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43256 (local address 127.0.203.3:41499)
0620 19:07:56.587474 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:56.587479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:56.587598 (+ 119us) server_negotiation.cc:407] Connection header received
0620 19:07:56.587731 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:56.587733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:56.587750 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:56.587801 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:56.588044 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.588802 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.590111 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.590724 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.590758 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.590797 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:56.591003 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 19:07:56.591039 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260620 19:07:56.591122 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.587245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:41499 (local address 127.0.0.1:43256)
0620 19:07:56.587494 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:56.587534 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:56.587623 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:56.587890 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:56.587891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:56.587893 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:56.587955 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.587957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.588883 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.588884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.590014 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.590017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.590844 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.590845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.590897 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.590905 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 19:07:56.590989 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:07:56.591039 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":61,"threads_started":1}
I20260620 19:07:56.591323 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.587451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:40833 (local address 127.0.0.1:35218)
0620 19:07:56.587593 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:56.587605 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:56.587654 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:56.588010 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:56.588011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:56.588013 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:56.588053 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.588054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.589298 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.589299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.590132 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.590134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.591075 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.591076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.591163 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.591169 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:07:56.591201 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 19:07:56.591244 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260620 19:07:56.591610 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.587270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:45187 (local address 127.0.0.1:45036)
0620 19:07:56.587553 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:56.587563 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:56.587623 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:56.587878 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:56.587882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:56.587884 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:56.587952 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.587954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.589260 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.589261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.590347 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.590349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.591324 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.591325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.591465 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.591469 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:07:56.591518 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:07:56.591554 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":81,"threads_started":1}
I20260620 19:07:56.591670 3042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.587660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35218 (local address 127.0.203.1:40833)
0620 19:07:56.587820 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:56.587824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:56.587834 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:07:56.587872 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:56.587874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:56.587888 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:56.588130 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:56.588192 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.589239 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.590232 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.590972 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.591439 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.591512 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:56.591539 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 19:07:56.591574 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":75,"threads_started":1}
I20260620 19:07:56.591923 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.587361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45036 (local address 127.0.203.2:45187)
0620 19:07:56.587539 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:56.587543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:56.587709 (+ 166us) server_negotiation.cc:407] Connection header received
0620 19:07:56.587758 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:56.587760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:56.587778 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:56.587871 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:56.588039 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.589165 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.590426 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.591166 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.591227 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.591292 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:56.591534 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 19:07:56.591578 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":77,"threads_started":1}
I20260620 19:07:56.592283 2953 tablet_service.cc:1467] Processing CreateTablet for tablet 175e6b8747f24ac29e51ee442a2c6423 (DEFAULT_TABLE table=test-table [id=0e4bf7abd3564d9eadf85a7ddd501f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:56.592332 2674 tablet_service.cc:1467] Processing CreateTablet for tablet 175e6b8747f24ac29e51ee442a2c6423 (DEFAULT_TABLE table=test-table [id=0e4bf7abd3564d9eadf85a7ddd501f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:56.592437 2814 tablet_service.cc:1467] Processing CreateTablet for tablet 175e6b8747f24ac29e51ee442a2c6423 (DEFAULT_TABLE table=test-table [id=0e4bf7abd3564d9eadf85a7ddd501f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:56.592625 2953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175e6b8747f24ac29e51ee442a2c6423. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:56.592684 2674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175e6b8747f24ac29e51ee442a2c6423. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:56.592697 2814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175e6b8747f24ac29e51ee442a2c6423. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:56.595084 3043 tablet_bootstrap.cc:492] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8: Bootstrap starting.
I20260620 19:07:56.596025 3043 tablet_bootstrap.cc:654] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:56.596335 3044 tablet_bootstrap.cc:492] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3: Bootstrap starting.
I20260620 19:07:56.596405 3043 log.cc:824] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:56.596560 3045 tablet_bootstrap.cc:492] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516: Bootstrap starting.
I20260620 19:07:56.597158 3045 tablet_bootstrap.cc:654] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:56.597249 3043 tablet_bootstrap.cc:492] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8: No bootstrap required, opened a new log
I20260620 19:07:56.597328 3043 ts_tablet_manager.cc:1397] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:07:56.597497 3045 log.cc:824] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:56.598235 3045 tablet_bootstrap.cc:492] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516: No bootstrap required, opened a new log
I20260620 19:07:56.598308 3045 ts_tablet_manager.cc:1397] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:07:56.598325 3044 tablet_bootstrap.cc:654] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:56.598634 3044 log.cc:824] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3: Log is configured to *not* fsync() on all Append() calls
I20260620 19:07:56.599478 3044 tablet_bootstrap.cc:492] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3: No bootstrap required, opened a new log
I20260620 19:07:56.599555 3044 ts_tablet_manager.cc:1397] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 19:07:56.599555 3043 raft_consensus.cc:348] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } }
I20260620 19:07:56.599730 3043 raft_consensus.cc:374] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:56.599774 3043 raft_consensus.cc:729] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 068176828f894b1599b47b45654b36c8, State: Initialized, Role: FOLLOWER
I20260620 19:07:56.599924 3043 consensus_queue.cc:260] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } }
I20260620 19:07:56.600199 3043 ts_tablet_manager.cc:1428] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:07:56.600329 3020 heartbeater.cc:503] Master 127.0.203.62:40855 was elected leader, sending a full tablet report...
I20260620 19:07:56.600281 3045 raft_consensus.cc:348] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } }
I20260620 19:07:56.600459 3045 raft_consensus.cc:374] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:56.600502 3045 raft_consensus.cc:729] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f315f8c5ed9f4fe38f59de5c7f238516, State: Initialized, Role: FOLLOWER
I20260620 19:07:56.600611 3045 consensus_queue.cc:260] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } }
I20260620 19:07:56.600886 2880 heartbeater.cc:503] Master 127.0.203.62:40855 was elected leader, sending a full tablet report...
I20260620 19:07:56.601142 3045 ts_tablet_manager.cc:1428] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:07:56.602058 3044 raft_consensus.cc:348] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } }
I20260620 19:07:56.602209 3044 raft_consensus.cc:374] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:56.602241 3044 raft_consensus.cc:729] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03fee9c5de834f9695a7976cb0503ca3, State: Initialized, Role: FOLLOWER
I20260620 19:07:56.602355 3044 consensus_queue.cc:260] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } }
I20260620 19:07:56.602566 3044 ts_tablet_manager.cc:1428] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:07:56.602643 2740 heartbeater.cc:503] Master 127.0.203.62:40855 was elected leader, sending a full tablet report...
W20260620 19:07:56.617600 2881 tablet.cc:2369] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:07:56.701059 2741 tablet.cc:2369] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:07:56.791086 3051 raft_consensus.cc:482] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:07:56.791276 3051 raft_consensus.cc:504] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } }
I20260620 19:07:56.791708 3051 leader_election.cc:288] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 068176828f894b1599b47b45654b36c8 (127.0.203.3:41499), f315f8c5ed9f4fe38f59de5c7f238516 (127.0.203.2:45187)
I20260620 19:07:56.794612 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.791988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:33379 (local address 127.0.203.3:41499)
0620 19:07:56.792050 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:56.792053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:56.792125 (+ 72us) server_negotiation.cc:407] Connection header received
0620 19:07:56.792194 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:56.792195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:56.792206 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:56.792275 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:56.792417 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.792897 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.793884 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.794242 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.794340 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.794350 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:56.794505 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 19:07:56.794546 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:07:56.794636 3052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.791909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:41499 (local address 127.0.203.1:33379)
0620 19:07:56.792034 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:56.792047 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:56.792133 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:56.792279 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:56.792280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:56.792282 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:56.792338 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.792340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.792978 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.792979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.793777 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.793779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.794351 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.794352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.794426 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.794430 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:07:56.794498 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 19:07:56.794540 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":35,"thread_start_us":58,"threads_started":1}
I20260620 19:07:56.794857 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.792133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:39997 (local address 127.0.203.2:45187)
0620 19:07:56.792180 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:56.792182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:56.792192 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:07:56.792222 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:56.792223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:56.792235 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:56.792338 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:56.792510 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.793021 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.794003 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.794411 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.794506 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.794515 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:56.794707 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 19:07:56.794769 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 19:07:56.794946 2974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175e6b8747f24ac29e51ee442a2c6423" candidate_uuid: "03fee9c5de834f9695a7976cb0503ca3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "068176828f894b1599b47b45654b36c8" is_pre_election: true
I20260620 19:07:56.795073 3053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.791968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:45187 (local address 127.0.203.1:39997)
0620 19:07:56.792091 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:56.792104 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:56.792161 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:56.792353 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:56.792354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:56.792356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:56.792432 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.792434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.793116 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.793117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.793909 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.793910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.794509 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.794510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.794592 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.794596 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:07:56.794939 (+ 343us) client_negotiation.cc:240] Negotiation successful
0620 19:07:56.794986 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260620 19:07:56.795138 2974 raft_consensus.cc:2454] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03fee9c5de834f9695a7976cb0503ca3 in term 0.
I20260620 19:07:56.795403 2627 leader_election.cc:302] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03fee9c5de834f9695a7976cb0503ca3, 068176828f894b1599b47b45654b36c8; no voters:
I20260620 19:07:56.795428 2834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175e6b8747f24ac29e51ee442a2c6423" candidate_uuid: "03fee9c5de834f9695a7976cb0503ca3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" is_pre_election: true
I20260620 19:07:56.795552 3051 raft_consensus.cc:2781] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:07:56.795630 3051 raft_consensus.cc:482] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:07:56.795655 3051 raft_consensus.cc:3037] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:56.795608 2834 raft_consensus.cc:2454] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03fee9c5de834f9695a7976cb0503ca3 in term 0.
I20260620 19:07:56.796413 3051 raft_consensus.cc:504] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } }
I20260620 19:07:56.796567 3051 leader_election.cc:288] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [CANDIDATE]: Term 1 election: Requested vote from peers 068176828f894b1599b47b45654b36c8 (127.0.203.3:41499), f315f8c5ed9f4fe38f59de5c7f238516 (127.0.203.2:45187)
I20260620 19:07:56.796751 2974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175e6b8747f24ac29e51ee442a2c6423" candidate_uuid: "03fee9c5de834f9695a7976cb0503ca3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "068176828f894b1599b47b45654b36c8"
I20260620 19:07:56.796841 2974 raft_consensus.cc:3037] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:56.796859 2834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175e6b8747f24ac29e51ee442a2c6423" candidate_uuid: "03fee9c5de834f9695a7976cb0503ca3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f315f8c5ed9f4fe38f59de5c7f238516"
I20260620 19:07:56.796943 2834 raft_consensus.cc:3037] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:56.797560 2974 raft_consensus.cc:2454] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03fee9c5de834f9695a7976cb0503ca3 in term 1.
I20260620 19:07:56.797721 2627 leader_election.cc:302] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03fee9c5de834f9695a7976cb0503ca3, 068176828f894b1599b47b45654b36c8; no voters:
I20260620 19:07:56.797806 3051 raft_consensus.cc:2781] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:07:56.797847 2834 raft_consensus.cc:2454] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03fee9c5de834f9695a7976cb0503ca3 in term 1.
I20260620 19:07:56.797966 3051 raft_consensus.cc:686] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [term 1 LEADER]: Becoming Leader. State: Replica: 03fee9c5de834f9695a7976cb0503ca3, State: Running, Role: LEADER
I20260620 19:07:56.798205 3051 consensus_queue.cc:237] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } }
I20260620 19:07:56.799094 2536 catalog_manager.cc:5495] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03fee9c5de834f9695a7976cb0503ca3 (127.0.203.1). New cstate: current_term: 1 leader_uuid: "03fee9c5de834f9695a7976cb0503ca3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } health_report { overall_health: HEALTHY } } }
W20260620 19:07:56.800812 3021 tablet.cc:2369] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 19:07:56 dist-test-slave-wd30 krb5kdc[2492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:07:56.843238 3061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.835901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:40855 (local address 127.0.0.1:37884)
0620 19:07:56.836134 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:56.836151 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:56.836239 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:56.836428 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:56.836429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:56.836743 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:07:56.836791 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.836793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.837500 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.837502 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.838058 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.838059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.838429 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.838430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.838487 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.839011 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:07:56.839016 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:07:56.840571 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:07:56.842132 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:56.842139 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:56.842141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:56.842342 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:56.842544 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:07:56.842545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:07:56.842545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:07:56.842599 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:07:56.842879 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:07:56.842885 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:07:56.842942 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 19:07:56.842999 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 19:07:56.843094 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":117,"threads_started":1}
I20260620 19:07:56.843338 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.835985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37884 (local address 127.0.203.62:40855)
0620 19:07:56.836099 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:56.836102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:56.836217 (+ 115us) server_negotiation.cc:407] Connection header received
0620 19:07:56.836338 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:56.836339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:56.836350 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:56.836424 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:56.836890 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.837404 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.838142 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.838343 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.838428 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.840661 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:56.840663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:56.840663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:56.840670 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:56.842049 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:56.842428 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:56.842430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:56.842430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:56.842482 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:56.842679 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:56.842680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:56.842681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:56.842812 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:56.842876 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:56.843037 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 19:07:56.843215 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":81}
I20260620 19:07:56.845487 2536 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:37884:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 19:07:56.845723 2536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:07:56.847330 2674 tablet_service.cc:1467] Processing CreateTablet for tablet d990bfd57e734136a4fe3cad5894130d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c79773b105ce4693a50e23eeb03a2ebb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:07:56.847353 2953 tablet_service.cc:1467] Processing CreateTablet for tablet d990bfd57e734136a4fe3cad5894130d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c79773b105ce4693a50e23eeb03a2ebb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:07:56.847379 2814 tablet_service.cc:1467] Processing CreateTablet for tablet d990bfd57e734136a4fe3cad5894130d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c79773b105ce4693a50e23eeb03a2ebb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:07:56.847512 2953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d990bfd57e734136a4fe3cad5894130d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:56.847512 2674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d990bfd57e734136a4fe3cad5894130d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:56.847577 2814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d990bfd57e734136a4fe3cad5894130d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:56.849041 3044 tablet_bootstrap.cc:492] T d990bfd57e734136a4fe3cad5894130d P 03fee9c5de834f9695a7976cb0503ca3: Bootstrap starting.
I20260620 19:07:56.849040 3045 tablet_bootstrap.cc:492] T d990bfd57e734136a4fe3cad5894130d P f315f8c5ed9f4fe38f59de5c7f238516: Bootstrap starting.
I20260620 19:07:56.849450 3044 tablet_bootstrap.cc:654] T d990bfd57e734136a4fe3cad5894130d P 03fee9c5de834f9695a7976cb0503ca3: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:56.849738 3045 tablet_bootstrap.cc:654] T d990bfd57e734136a4fe3cad5894130d P f315f8c5ed9f4fe38f59de5c7f238516: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:56.850106 3043 tablet_bootstrap.cc:492] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8: Bootstrap starting.
I20260620 19:07:56.850338 3044 tablet_bootstrap.cc:492] T d990bfd57e734136a4fe3cad5894130d P 03fee9c5de834f9695a7976cb0503ca3: No bootstrap required, opened a new log
I20260620 19:07:56.850379 3044 ts_tablet_manager.cc:1397] T d990bfd57e734136a4fe3cad5894130d P 03fee9c5de834f9695a7976cb0503ca3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:07:56.850397 3045 tablet_bootstrap.cc:492] T d990bfd57e734136a4fe3cad5894130d P f315f8c5ed9f4fe38f59de5c7f238516: No bootstrap required, opened a new log
I20260620 19:07:56.850451 3045 ts_tablet_manager.cc:1397] T d990bfd57e734136a4fe3cad5894130d P f315f8c5ed9f4fe38f59de5c7f238516: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:07:56.850531 3044 raft_consensus.cc:348] T d990bfd57e734136a4fe3cad5894130d P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:56.850603 3044 raft_consensus.cc:374] T d990bfd57e734136a4fe3cad5894130d P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:56.850625 3044 raft_consensus.cc:729] T d990bfd57e734136a4fe3cad5894130d P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03fee9c5de834f9695a7976cb0503ca3, State: Initialized, Role: FOLLOWER
I20260620 19:07:56.850586 3045 raft_consensus.cc:348] T d990bfd57e734136a4fe3cad5894130d P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:56.850625 3043 tablet_bootstrap.cc:654] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:56.850649 3045 raft_consensus.cc:374] T d990bfd57e734136a4fe3cad5894130d P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:56.850672 3045 raft_consensus.cc:729] T d990bfd57e734136a4fe3cad5894130d P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f315f8c5ed9f4fe38f59de5c7f238516, State: Initialized, Role: FOLLOWER
I20260620 19:07:56.850675 3044 consensus_queue.cc:260] T d990bfd57e734136a4fe3cad5894130d P 03fee9c5de834f9695a7976cb0503ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:56.850714 3045 consensus_queue.cc:260] T d990bfd57e734136a4fe3cad5894130d P f315f8c5ed9f4fe38f59de5c7f238516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:56.850775 3044 ts_tablet_manager.cc:1428] T d990bfd57e734136a4fe3cad5894130d P 03fee9c5de834f9695a7976cb0503ca3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:56.850793 3045 ts_tablet_manager.cc:1428] T d990bfd57e734136a4fe3cad5894130d P f315f8c5ed9f4fe38f59de5c7f238516: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:56.850852 3051 tablet_replica.cc:439] T d990bfd57e734136a4fe3cad5894130d P 03fee9c5de834f9695a7976cb0503ca3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } } }
I20260620 19:07:56.850967 3051 tablet_replica.cc:442] T d990bfd57e734136a4fe3cad5894130d P 03fee9c5de834f9695a7976cb0503ca3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:07:56.851095 3050 tablet_replica.cc:439] T d990bfd57e734136a4fe3cad5894130d P f315f8c5ed9f4fe38f59de5c7f238516: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } } }
I20260620 19:07:56.851235 3050 tablet_replica.cc:442] T d990bfd57e734136a4fe3cad5894130d P f315f8c5ed9f4fe38f59de5c7f238516: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:07:56.851958 3043 tablet_bootstrap.cc:492] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8: No bootstrap required, opened a new log
I20260620 19:07:56.852000 3043 ts_tablet_manager.cc:1397] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:07:56.852114 3043 raft_consensus.cc:348] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:56.852172 3043 raft_consensus.cc:374] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:56.852186 3043 raft_consensus.cc:729] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 068176828f894b1599b47b45654b36c8, State: Initialized, Role: FOLLOWER
I20260620 19:07:56.852246 3043 consensus_queue.cc:260] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:56.852339 3043 ts_tablet_manager.cc:1428] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:56.852407 3049 tablet_replica.cc:439] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } } }
I20260620 19:07:56.852483 3049 tablet_replica.cc:442] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:07:56.933633 3049 raft_consensus.cc:482] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:07:56.933733 3049 raft_consensus.cc:504] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:56.934125 3049 leader_election.cc:288] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03fee9c5de834f9695a7976cb0503ca3 (127.0.203.1:40833), f315f8c5ed9f4fe38f59de5c7f238516 (127.0.203.2:45187)
I20260620 19:07:56.937680 2913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.934319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:40833 (local address 127.0.203.3:58867)
0620 19:07:56.934382 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:56.934396 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:56.934446 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:56.934718 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:56.934720 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:56.934723 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:56.934802 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.934804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.935610 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.935611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.936836 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.936839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.937396 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.937396 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.937473 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.937476 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:07:56.937540 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 19:07:56.937583 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":1}
I20260620 19:07:56.937664 3042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.934410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:58867 (local address 127.0.203.1:40833)
0620 19:07:56.934473 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:56.934475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:56.934489 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:07:56.934522 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:56.934523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:56.934536 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:56.934640 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:56.934892 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.935506 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.936927 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.937310 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.937397 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.937409 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:56.937560 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 19:07:56.937602 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:07:56.938069 2694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d990bfd57e734136a4fe3cad5894130d" candidate_uuid: "068176828f894b1599b47b45654b36c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03fee9c5de834f9695a7976cb0503ca3" is_pre_election: true
I20260620 19:07:56.938206 2694 raft_consensus.cc:2454] T d990bfd57e734136a4fe3cad5894130d P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 068176828f894b1599b47b45654b36c8 in term 0.
I20260620 19:07:56.938382 2908 leader_election.cc:302] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03fee9c5de834f9695a7976cb0503ca3, 068176828f894b1599b47b45654b36c8; no voters:
I20260620 19:07:56.938547 3049 raft_consensus.cc:2781] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:07:56.938606 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.934552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:41517 (local address 127.0.203.2:45187)
0620 19:07:56.934616 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:56.934618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:56.934718 (+ 100us) server_negotiation.cc:407] Connection header received
0620 19:07:56.934749 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:56.934750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:56.934759 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:56.934870 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:56.935756 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.936547 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.937582 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.938222 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.938313 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.938325 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:56.938487 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 19:07:56.938530 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:07:56.938621 3049 raft_consensus.cc:482] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:07:56.938657 3049 raft_consensus.cc:3037] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:56.938690 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.934469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:45187 (local address 127.0.203.3:41517)
0620 19:07:56.934644 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:56.934663 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:56.934711 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:56.935597 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:56.935598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:56.935600 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:56.935668 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.935670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.936637 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.936638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.937470 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.937472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.938312 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.938313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.938396 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.938400 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:07:56.938479 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 19:07:56.938524 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":4,"thread_start_us":110,"threads_started":1}
I20260620 19:07:56.938908 2834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d990bfd57e734136a4fe3cad5894130d" candidate_uuid: "068176828f894b1599b47b45654b36c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" is_pre_election: true
I20260620 19:07:56.938997 2834 raft_consensus.cc:2454] T d990bfd57e734136a4fe3cad5894130d P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 068176828f894b1599b47b45654b36c8 in term 0.
I20260620 19:07:56.939576 3049 raft_consensus.cc:504] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:56.939723 3049 leader_election.cc:288] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8 [CANDIDATE]: Term 1 election: Requested vote from peers 03fee9c5de834f9695a7976cb0503ca3 (127.0.203.1:40833), f315f8c5ed9f4fe38f59de5c7f238516 (127.0.203.2:45187)
I20260620 19:07:56.939898 2834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d990bfd57e734136a4fe3cad5894130d" candidate_uuid: "068176828f894b1599b47b45654b36c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f315f8c5ed9f4fe38f59de5c7f238516"
I20260620 19:07:56.939993 2834 raft_consensus.cc:3037] T d990bfd57e734136a4fe3cad5894130d P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:56.940009 2694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d990bfd57e734136a4fe3cad5894130d" candidate_uuid: "068176828f894b1599b47b45654b36c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03fee9c5de834f9695a7976cb0503ca3"
I20260620 19:07:56.940088 2694 raft_consensus.cc:3037] T d990bfd57e734136a4fe3cad5894130d P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:56.940780 2834 raft_consensus.cc:2454] T d990bfd57e734136a4fe3cad5894130d P f315f8c5ed9f4fe38f59de5c7f238516 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 068176828f894b1599b47b45654b36c8 in term 1.
I20260620 19:07:56.940950 2908 leader_election.cc:302] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 068176828f894b1599b47b45654b36c8, f315f8c5ed9f4fe38f59de5c7f238516; no voters:
I20260620 19:07:56.941066 3049 raft_consensus.cc:2781] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:07:56.941094 2694 raft_consensus.cc:2454] T d990bfd57e734136a4fe3cad5894130d P 03fee9c5de834f9695a7976cb0503ca3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 068176828f894b1599b47b45654b36c8 in term 1.
I20260620 19:07:56.941256 3049 raft_consensus.cc:686] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8 [term 1 LEADER]: Becoming Leader. State: Replica: 068176828f894b1599b47b45654b36c8, State: Running, Role: LEADER
I20260620 19:07:56.941388 3049 consensus_queue.cc:237] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:56.941887 3066 tablet_replica.cc:439] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8: TxnStatusTablet state changed. Reason: New leader 068176828f894b1599b47b45654b36c8. Latest consensus state: current_term: 1 leader_uuid: "068176828f894b1599b47b45654b36c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } } }
I20260620 19:07:56.941999 3066 tablet_replica.cc:442] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:07:56.942198 3068 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 19:07:56.942360 3068 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 19:07:56.942281 2536 catalog_manager.cc:5495] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 068176828f894b1599b47b45654b36c8 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "068176828f894b1599b47b45654b36c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } health_report { overall_health: HEALTHY } } }
I20260620 19:07:56.969501 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.966587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.0.203.3:41499)
0620 19:07:56.966648 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:56.966650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:56.966668 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:07:56.966699 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:56.966700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:56.966715 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:56.966778 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:07:56.966946 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.967513 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.968716 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.968913 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.968964 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.969181 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:07:56.969272 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:07:56.969326 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:56.969407 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 19:07:56.969424 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:07:56.969547 3061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.966484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:41499 (local address 127.0.0.1:43266)
0620 19:07:56.966560 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:56.966572 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:56.966619 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:56.966813 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:56.966815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:56.966817 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:07:56.966873 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.966876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.967606 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.967607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.968633 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.968637 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.968990 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.968992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.969066 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.969084 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:07:56.969336 (+ 252us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:07:56.969343 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 19:07:56.969405 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:07:56.969456 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 19:07:56.971472 2694 raft_consensus.cc:1261] T d990bfd57e734136a4fe3cad5894130d P 03fee9c5de834f9695a7976cb0503ca3 [term 1 FOLLOWER]: Refusing update from remote peer 068176828f894b1599b47b45654b36c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:56.971483 2834 raft_consensus.cc:1261] T d990bfd57e734136a4fe3cad5894130d P f315f8c5ed9f4fe38f59de5c7f238516 [term 1 FOLLOWER]: Refusing update from remote peer 068176828f894b1599b47b45654b36c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:56.971822 3066 consensus_queue.cc:1035] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:56.971982 3049 consensus_queue.cc:1035] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:56.973145 3051 tablet_replica.cc:439] T d990bfd57e734136a4fe3cad5894130d P 03fee9c5de834f9695a7976cb0503ca3: TxnStatusTablet state changed. Reason: New leader 068176828f894b1599b47b45654b36c8. Latest consensus state: current_term: 1 leader_uuid: "068176828f894b1599b47b45654b36c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } } }
I20260620 19:07:56.973237 3051 tablet_replica.cc:442] T d990bfd57e734136a4fe3cad5894130d P 03fee9c5de834f9695a7976cb0503ca3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:07:56.973191 3050 tablet_replica.cc:439] T d990bfd57e734136a4fe3cad5894130d P f315f8c5ed9f4fe38f59de5c7f238516: TxnStatusTablet state changed. Reason: New leader 068176828f894b1599b47b45654b36c8. Latest consensus state: current_term: 1 leader_uuid: "068176828f894b1599b47b45654b36c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } } }
I20260620 19:07:56.973280 3050 tablet_replica.cc:442] T d990bfd57e734136a4fe3cad5894130d P f315f8c5ed9f4fe38f59de5c7f238516: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:07:56.974668 3049 tablet_replica.cc:439] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "068176828f894b1599b47b45654b36c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } } }
I20260620 19:07:56.974778 3049 tablet_replica.cc:442] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:07:56.975347 3071 mvcc.cc:205] Tried to move back new op lower bound from 7299000225671954432 to 7299000225552470016. Current Snapshot: MvccSnapshot[applied={T|T < 7299000225671954432 or (T in {7299000225671954432})}]
I20260620 19:07:56.975404 3050 tablet_replica.cc:439] T d990bfd57e734136a4fe3cad5894130d P f315f8c5ed9f4fe38f59de5c7f238516: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "068176828f894b1599b47b45654b36c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } } }
I20260620 19:07:56.975483 3050 tablet_replica.cc:442] T d990bfd57e734136a4fe3cad5894130d P f315f8c5ed9f4fe38f59de5c7f238516: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:07:56.975852 3049 tablet_replica.cc:439] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "068176828f894b1599b47b45654b36c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } } }
I20260620 19:07:56.975925 3049 tablet_replica.cc:442] T d990bfd57e734136a4fe3cad5894130d P 068176828f894b1599b47b45654b36c8: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:07:56.976181 3051 tablet_replica.cc:439] T d990bfd57e734136a4fe3cad5894130d P 03fee9c5de834f9695a7976cb0503ca3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "068176828f894b1599b47b45654b36c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } } }
I20260620 19:07:56.976284 3051 tablet_replica.cc:442] T d990bfd57e734136a4fe3cad5894130d P 03fee9c5de834f9695a7976cb0503ca3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:07:56.976361 3072 mvcc.cc:205] Tried to move back new op lower bound from 7299000225671954432 to 7299000225552470016. Current Snapshot: MvccSnapshot[applied={T|T < 7299000225671954432}]
I20260620 19:07:56.979610 3042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.976850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35222 (local address 127.0.203.1:40833)
0620 19:07:56.976886 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:56.976887 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:56.976896 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:07:56.976968 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:56.976969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:56.976986 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:56.977064 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:07:56.977141 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.977884 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.978645 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.978928 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.979001 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.979158 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:07:56.979277 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:07:56.979357 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:56.979509 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 19:07:56.979533 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:07:56.984462 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.981996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:37661 (local address 127.0.203.3:41499)
0620 19:07:56.982029 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:56.982031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:56.982168 (+ 137us) server_negotiation.cc:407] Connection header received
0620 19:07:56.982239 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:56.982240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:56.982249 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:56.982315 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:07:56.982460 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.982912 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.983768 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.983956 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.984024 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.984221 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:07:56.984256 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:07:56.984316 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:56.984384 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 19:07:56.984399 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:07:56.984509 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.981935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:41499 (local address 127.0.203.1:37661)
0620 19:07:56.982102 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:56.982111 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:56.982183 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:56.982317 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:56.982318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:56.982319 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:07:56.982391 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.982397 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.982994 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.982995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.983673 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.983675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.984036 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.984037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.984107 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.984119 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:07:56.984318 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:07:56.984320 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:07:56.984379 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 19:07:56.984418 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":70,"threads_started":1}
I20260620 19:07:56.986796 2834 raft_consensus.cc:1261] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516 [term 1 FOLLOWER]: Refusing update from remote peer 03fee9c5de834f9695a7976cb0503ca3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:56.986913 2974 raft_consensus.cc:1261] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8 [term 1 FOLLOWER]: Refusing update from remote peer 03fee9c5de834f9695a7976cb0503ca3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:56.987048 3054 consensus_queue.cc:1035] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:56.987190 3051 consensus_queue.cc:1035] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:56.989151 3073 mvcc.cc:205] Tried to move back new op lower bound from 7299000225736044544 to 7299000224966148096. Current Snapshot: MvccSnapshot[applied={T|T < 7299000225736044544}]
I20260620 19:07:56.995947 3026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.993044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:40833 (local address 127.0.203.3:60143)
0620 19:07:56.993106 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:56.993118 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:56.993161 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:56.993302 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:56.993303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:56.993304 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:07:56.993361 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.993362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.994205 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.994206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.995094 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:56.995096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.995589 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.995590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:56.995635 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.995649 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:07:56.995810 (+ 161us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:07:56.995812 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:07:56.995865 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 19:07:56.995898 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 19:07:56.995960 3042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:56.993092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:60143 (local address 127.0.203.1:40833)
0620 19:07:56.993134 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:56.993135 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:56.993147 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:07:56.993218 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:56.993219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:56.993229 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:56.993277 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:07:56.993423 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.994116 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.995217 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:56.995516 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:56.995586 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:56.995720 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:07:56.995750 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:07:56.995803 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:56.995869 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 19:07:56.995887 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:07:57.005744 2536 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:07:57.005877 2536 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:07:57.006707 2536 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b: Sending DeleteTablet for 3 replicas of tablet 175e6b8747f24ac29e51ee442a2c6423
I20260620 19:07:57.006999 2953 tablet_service.cc:1514] Processing DeleteTablet for tablet 175e6b8747f24ac29e51ee442a2c6423 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:56 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:43256
I20260620 19:07:57.007155 2814 tablet_service.cc:1514] Processing DeleteTablet for tablet 175e6b8747f24ac29e51ee442a2c6423 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:56 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:45036
I20260620 19:07:57.007265 3090 tablet_replica.cc:330] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8: stopping tablet replica
I20260620 19:07:57.007385 3090 raft_consensus.cc:2229] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:07:57.007391 2674 tablet_service.cc:1514] Processing DeleteTablet for tablet 175e6b8747f24ac29e51ee442a2c6423 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:56 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:35218
I20260620 19:07:57.007442 3090 raft_consensus.cc:2258] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:57.007591 3092 tablet_replica.cc:330] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3: stopping tablet replica
I20260620 19:07:57.007678 3092 raft_consensus.cc:2229] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:07:57.007773 3092 raft_consensus.cc:2258] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:57.008195 3090 ts_tablet_manager.cc:1905] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:57.008226 2536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:07:57.008384 2536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:07:57.009738 2953 tablet_service.cc:1467] Processing CreateTablet for tablet a81c809279e64c088b05b6a9df8bcb4f (DEFAULT_TABLE table=test-table [id=295eda90cdf74a45900b448682d36cc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:57.009907 2953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a81c809279e64c088b05b6a9df8bcb4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:57.010169 2674 tablet_service.cc:1467] Processing CreateTablet for tablet a81c809279e64c088b05b6a9df8bcb4f (DEFAULT_TABLE table=test-table [id=295eda90cdf74a45900b448682d36cc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:57.010293 2674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a81c809279e64c088b05b6a9df8bcb4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:57.011209 3043 tablet_bootstrap.cc:492] T a81c809279e64c088b05b6a9df8bcb4f P 068176828f894b1599b47b45654b36c8: Bootstrap starting.
I20260620 19:07:57.011627 3043 tablet_bootstrap.cc:654] T a81c809279e64c088b05b6a9df8bcb4f P 068176828f894b1599b47b45654b36c8: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:57.011828 2814 tablet_service.cc:1467] Processing CreateTablet for tablet a81c809279e64c088b05b6a9df8bcb4f (DEFAULT_TABLE table=test-table [id=295eda90cdf74a45900b448682d36cc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:57.011948 2814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a81c809279e64c088b05b6a9df8bcb4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:57.012158 3044 tablet_bootstrap.cc:492] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3: Bootstrap starting.
I20260620 19:07:57.012581 3044 tablet_bootstrap.cc:654] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:57.009918 3090 ts_tablet_manager.cc:1918] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:07:57.013021 3090 log.cc:1196] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/175e6b8747f24ac29e51ee442a2c6423
I20260620 19:07:57.013286 3090 ts_tablet_manager.cc:1939] T 175e6b8747f24ac29e51ee442a2c6423 P 068176828f894b1599b47b45654b36c8: Deleting consensus metadata
I20260620 19:07:57.013319 3045 tablet_bootstrap.cc:492] T a81c809279e64c088b05b6a9df8bcb4f P f315f8c5ed9f4fe38f59de5c7f238516: Bootstrap starting.
I20260620 19:07:57.013664 3045 tablet_bootstrap.cc:654] T a81c809279e64c088b05b6a9df8bcb4f P f315f8c5ed9f4fe38f59de5c7f238516: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:57.014011 3092 ts_tablet_manager.cc:1905] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:57.014030 2523 catalog_manager.cc:4841] TS 068176828f894b1599b47b45654b36c8 (127.0.203.3:41499): tablet 175e6b8747f24ac29e51ee442a2c6423 (table test-table [id=0e4bf7abd3564d9eadf85a7ddd501f63]) successfully deleted
I20260620 19:07:57.014941 3091 tablet_replica.cc:330] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516: stopping tablet replica
I20260620 19:07:57.015142 3091 raft_consensus.cc:2229] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:07:57.015312 3091 raft_consensus.cc:2258] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:57.015812 3091 ts_tablet_manager.cc:1905] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:57.017750 3091 ts_tablet_manager.cc:1918] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:07:57.017879 3091 log.cc:1196] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/175e6b8747f24ac29e51ee442a2c6423
I20260620 19:07:57.018095 3044 tablet_bootstrap.cc:492] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3: No bootstrap required, opened a new log
I20260620 19:07:57.018152 3044 ts_tablet_manager.cc:1397] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 19:07:57.018174 3045 tablet_bootstrap.cc:492] T a81c809279e64c088b05b6a9df8bcb4f P f315f8c5ed9f4fe38f59de5c7f238516: No bootstrap required, opened a new log
I20260620 19:07:57.018214 3045 ts_tablet_manager.cc:1397] T a81c809279e64c088b05b6a9df8bcb4f P f315f8c5ed9f4fe38f59de5c7f238516: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 19:07:57.018329 3044 raft_consensus.cc:348] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:57.018369 3045 raft_consensus.cc:348] T a81c809279e64c088b05b6a9df8bcb4f P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:57.018429 3045 raft_consensus.cc:374] T a81c809279e64c088b05b6a9df8bcb4f P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:57.018429 3044 raft_consensus.cc:374] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:57.018451 3045 raft_consensus.cc:729] T a81c809279e64c088b05b6a9df8bcb4f P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f315f8c5ed9f4fe38f59de5c7f238516, State: Initialized, Role: FOLLOWER
I20260620 19:07:57.018451 3044 raft_consensus.cc:729] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03fee9c5de834f9695a7976cb0503ca3, State: Initialized, Role: FOLLOWER
I20260620 19:07:57.018494 3045 consensus_queue.cc:260] T a81c809279e64c088b05b6a9df8bcb4f P f315f8c5ed9f4fe38f59de5c7f238516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:57.018494 3044 consensus_queue.cc:260] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:57.018579 3045 ts_tablet_manager.cc:1428] T a81c809279e64c088b05b6a9df8bcb4f P f315f8c5ed9f4fe38f59de5c7f238516: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:57.018580 3044 ts_tablet_manager.cc:1428] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:07:57.019769 3091 ts_tablet_manager.cc:1939] T 175e6b8747f24ac29e51ee442a2c6423 P f315f8c5ed9f4fe38f59de5c7f238516: Deleting consensus metadata
I20260620 19:07:57.019881 3092 ts_tablet_manager.cc:1918] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:07:57.019959 3092 log.cc:1196] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-0/wal/wals/175e6b8747f24ac29e51ee442a2c6423
I20260620 19:07:57.020227 3092 ts_tablet_manager.cc:1939] T 175e6b8747f24ac29e51ee442a2c6423 P 03fee9c5de834f9695a7976cb0503ca3: Deleting consensus metadata
I20260620 19:07:57.020411 2524 catalog_manager.cc:4841] TS f315f8c5ed9f4fe38f59de5c7f238516 (127.0.203.2:45187): tablet 175e6b8747f24ac29e51ee442a2c6423 (table test-table [id=0e4bf7abd3564d9eadf85a7ddd501f63]) successfully deleted
I20260620 19:07:57.020630 2524 catalog_manager.cc:4841] TS 03fee9c5de834f9695a7976cb0503ca3 (127.0.203.1:40833): tablet 175e6b8747f24ac29e51ee442a2c6423 (table test-table [id=0e4bf7abd3564d9eadf85a7ddd501f63]) successfully deleted
I20260620 19:07:57.021085 3043 tablet_bootstrap.cc:492] T a81c809279e64c088b05b6a9df8bcb4f P 068176828f894b1599b47b45654b36c8: No bootstrap required, opened a new log
I20260620 19:07:57.021140 3043 ts_tablet_manager.cc:1397] T a81c809279e64c088b05b6a9df8bcb4f P 068176828f894b1599b47b45654b36c8: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260620 19:07:57.021265 3043 raft_consensus.cc:348] T a81c809279e64c088b05b6a9df8bcb4f P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:57.021347 3043 raft_consensus.cc:374] T a81c809279e64c088b05b6a9df8bcb4f P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:57.021369 3043 raft_consensus.cc:729] T a81c809279e64c088b05b6a9df8bcb4f P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 068176828f894b1599b47b45654b36c8, State: Initialized, Role: FOLLOWER
I20260620 19:07:57.021422 3043 consensus_queue.cc:260] T a81c809279e64c088b05b6a9df8bcb4f P 068176828f894b1599b47b45654b36c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:57.021519 3043 ts_tablet_manager.cc:1428] T a81c809279e64c088b05b6a9df8bcb4f P 068176828f894b1599b47b45654b36c8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:57.226716 3088 raft_consensus.cc:482] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:07:57.226888 3088 raft_consensus.cc:504] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:57.227245 3088 leader_election.cc:288] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f315f8c5ed9f4fe38f59de5c7f238516 (127.0.203.2:45187), 068176828f894b1599b47b45654b36c8 (127.0.203.3:41499)
I20260620 19:07:57.227483 2974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c809279e64c088b05b6a9df8bcb4f" candidate_uuid: "03fee9c5de834f9695a7976cb0503ca3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "068176828f894b1599b47b45654b36c8" is_pre_election: true
I20260620 19:07:57.227502 2834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c809279e64c088b05b6a9df8bcb4f" candidate_uuid: "03fee9c5de834f9695a7976cb0503ca3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" is_pre_election: true
I20260620 19:07:57.227607 2834 raft_consensus.cc:2454] T a81c809279e64c088b05b6a9df8bcb4f P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03fee9c5de834f9695a7976cb0503ca3 in term 0.
I20260620 19:07:57.227612 2974 raft_consensus.cc:2454] T a81c809279e64c088b05b6a9df8bcb4f P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03fee9c5de834f9695a7976cb0503ca3 in term 0.
I20260620 19:07:57.228006 2628 leader_election.cc:302] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03fee9c5de834f9695a7976cb0503ca3, f315f8c5ed9f4fe38f59de5c7f238516; no voters:
I20260620 19:07:57.228191 3088 raft_consensus.cc:2781] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:07:57.228274 3088 raft_consensus.cc:482] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:07:57.228304 3088 raft_consensus.cc:3037] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:57.229112 3088 raft_consensus.cc:504] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:57.229332 3088 leader_election.cc:288] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [CANDIDATE]: Term 1 election: Requested vote from peers f315f8c5ed9f4fe38f59de5c7f238516 (127.0.203.2:45187), 068176828f894b1599b47b45654b36c8 (127.0.203.3:41499)
I20260620 19:07:57.229564 2974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c809279e64c088b05b6a9df8bcb4f" candidate_uuid: "03fee9c5de834f9695a7976cb0503ca3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "068176828f894b1599b47b45654b36c8"
I20260620 19:07:57.229626 2834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c809279e64c088b05b6a9df8bcb4f" candidate_uuid: "03fee9c5de834f9695a7976cb0503ca3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f315f8c5ed9f4fe38f59de5c7f238516"
I20260620 19:07:57.229697 2974 raft_consensus.cc:3037] T a81c809279e64c088b05b6a9df8bcb4f P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:57.229735 2834 raft_consensus.cc:3037] T a81c809279e64c088b05b6a9df8bcb4f P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:57.230340 2834 raft_consensus.cc:2454] T a81c809279e64c088b05b6a9df8bcb4f P f315f8c5ed9f4fe38f59de5c7f238516 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03fee9c5de834f9695a7976cb0503ca3 in term 1.
I20260620 19:07:57.230446 2974 raft_consensus.cc:2454] T a81c809279e64c088b05b6a9df8bcb4f P 068176828f894b1599b47b45654b36c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03fee9c5de834f9695a7976cb0503ca3 in term 1.
I20260620 19:07:57.230543 2628 leader_election.cc:302] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03fee9c5de834f9695a7976cb0503ca3, f315f8c5ed9f4fe38f59de5c7f238516; no voters:
I20260620 19:07:57.230664 3088 raft_consensus.cc:2781] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:07:57.230718 3088 raft_consensus.cc:686] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [term 1 LEADER]: Becoming Leader. State: Replica: 03fee9c5de834f9695a7976cb0503ca3, State: Running, Role: LEADER
I20260620 19:07:57.230778 3088 consensus_queue.cc:237] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:57.231406 2537 catalog_manager.cc:5495] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03fee9c5de834f9695a7976cb0503ca3 (127.0.203.1). New cstate: current_term: 1 leader_uuid: "03fee9c5de834f9695a7976cb0503ca3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } health_report { overall_health: UNKNOWN } } }
I20260620 19:07:57.248072 2834 raft_consensus.cc:1261] T a81c809279e64c088b05b6a9df8bcb4f P f315f8c5ed9f4fe38f59de5c7f238516 [term 1 FOLLOWER]: Refusing update from remote peer 03fee9c5de834f9695a7976cb0503ca3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:57.248358 3088 consensus_queue.cc:1035] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:57.248747 2974 raft_consensus.cc:1261] T a81c809279e64c088b05b6a9df8bcb4f P 068176828f894b1599b47b45654b36c8 [term 1 FOLLOWER]: Refusing update from remote peer 03fee9c5de834f9695a7976cb0503ca3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:57.250494 3088 consensus_queue.cc:1035] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:07:57.253240 3072 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 19:07:58.772605 2537 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:07:58.772727 2537 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:07:58.773526 2537 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b: Sending DeleteTablet for 3 replicas of tablet a81c809279e64c088b05b6a9df8bcb4f
I20260620 19:07:58.773835 2814 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c809279e64c088b05b6a9df8bcb4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:58 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:45036
I20260620 19:07:58.773836 2674 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c809279e64c088b05b6a9df8bcb4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:58 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:35218
I20260620 19:07:58.774114 3122 tablet_replica.cc:330] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3: stopping tablet replica
I20260620 19:07:58.774183 3122 raft_consensus.cc:2229] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:07:58.774251 3122 raft_consensus.cc:2258] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:58.774370 3121 tablet_replica.cc:330] T a81c809279e64c088b05b6a9df8bcb4f P f315f8c5ed9f4fe38f59de5c7f238516: stopping tablet replica
I20260620 19:07:58.774428 3121 raft_consensus.cc:2229] T a81c809279e64c088b05b6a9df8bcb4f P f315f8c5ed9f4fe38f59de5c7f238516 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:07:58.774467 3121 raft_consensus.cc:2258] T a81c809279e64c088b05b6a9df8bcb4f P f315f8c5ed9f4fe38f59de5c7f238516 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:58.774596 3122 ts_tablet_manager.cc:1905] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:58.774690 3121 ts_tablet_manager.cc:1905] T a81c809279e64c088b05b6a9df8bcb4f P f315f8c5ed9f4fe38f59de5c7f238516: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:58.775205 2953 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c809279e64c088b05b6a9df8bcb4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:58 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:43256
I20260620 19:07:58.776543 3124 tablet_replica.cc:330] T a81c809279e64c088b05b6a9df8bcb4f P 068176828f894b1599b47b45654b36c8: stopping tablet replica
I20260620 19:07:58.776623 3124 raft_consensus.cc:2229] T a81c809279e64c088b05b6a9df8bcb4f P 068176828f894b1599b47b45654b36c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:07:58.776674 3124 raft_consensus.cc:2258] T a81c809279e64c088b05b6a9df8bcb4f P 068176828f894b1599b47b45654b36c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:58.776891 3124 ts_tablet_manager.cc:1905] T a81c809279e64c088b05b6a9df8bcb4f P 068176828f894b1599b47b45654b36c8: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 19:07:58 dist-test-slave-wd30 krb5kdc[2492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982478, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260620 19:07:58.797040 3122 ts_tablet_manager.cc:1918] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:07:58.797129 3122 log.cc:1196] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-0/wal/wals/a81c809279e64c088b05b6a9df8bcb4f
I20260620 19:07:58.797360 3122 ts_tablet_manager.cc:1939] T a81c809279e64c088b05b6a9df8bcb4f P 03fee9c5de834f9695a7976cb0503ca3: Deleting consensus metadata
I20260620 19:07:58.798099 3121 ts_tablet_manager.cc:1918] T a81c809279e64c088b05b6a9df8bcb4f P f315f8c5ed9f4fe38f59de5c7f238516: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:07:58.798123 3124 ts_tablet_manager.cc:1918] T a81c809279e64c088b05b6a9df8bcb4f P 068176828f894b1599b47b45654b36c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:07:58.798188 3124 log.cc:1196] T a81c809279e64c088b05b6a9df8bcb4f P 068176828f894b1599b47b45654b36c8: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/a81c809279e64c088b05b6a9df8bcb4f
I20260620 19:07:58.798185 3121 log.cc:1196] T a81c809279e64c088b05b6a9df8bcb4f P f315f8c5ed9f4fe38f59de5c7f238516: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/a81c809279e64c088b05b6a9df8bcb4f
I20260620 19:07:58.798422 3124 ts_tablet_manager.cc:1939] T a81c809279e64c088b05b6a9df8bcb4f P 068176828f894b1599b47b45654b36c8: Deleting consensus metadata
I20260620 19:07:58.798445 3121 ts_tablet_manager.cc:1939] T a81c809279e64c088b05b6a9df8bcb4f P f315f8c5ed9f4fe38f59de5c7f238516: Deleting consensus metadata
I20260620 19:07:58.804062 2524 catalog_manager.cc:4841] TS 03fee9c5de834f9695a7976cb0503ca3 (127.0.203.1:40833): tablet a81c809279e64c088b05b6a9df8bcb4f (table test-table [id=295eda90cdf74a45900b448682d36cc5]) successfully deleted
I20260620 19:07:58.804086 2523 catalog_manager.cc:4841] TS 068176828f894b1599b47b45654b36c8 (127.0.203.3:41499): tablet a81c809279e64c088b05b6a9df8bcb4f (table test-table [id=295eda90cdf74a45900b448682d36cc5]) successfully deleted
I20260620 19:07:58.804195 2524 catalog_manager.cc:4841] TS f315f8c5ed9f4fe38f59de5c7f238516 (127.0.203.2:45187): tablet a81c809279e64c088b05b6a9df8bcb4f (table test-table [id=295eda90cdf74a45900b448682d36cc5]) successfully deleted
Jun 20 19:07:58 dist-test-slave-wd30 krb5kdc[2492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982478, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:07:58.809360 3131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:58.796352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37892 (local address 127.0.203.62:40855)
0620 19:07:58.796514 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:58.796516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:58.796527 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:07:58.796590 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:58.796591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:58.796605 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:58.796679 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:07:58.797385 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:58.798059 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:58.799237 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:58.799440 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:58.799504 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:58.806480 (+ 6976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:07:58.806482 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:07:58.806483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:07:58.806491 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:07:58.808123 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:58.808530 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:58.808531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:58.808531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:58.808595 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:07:58.808787 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:07:58.808787 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:07:58.808788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:07:58.808908 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:07:58.808995 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:58.809166 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 19:07:58.809242 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260620 19:07:58.811110 2537 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:07:58.811350 2537 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:07:58.812839 2674 tablet_service.cc:1467] Processing CreateTablet for tablet 78bc21a85343470099406b858ea13583 (DEFAULT_TABLE table=test-table [id=7093241c62ec405aae69104d59a9fc8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:58.812839 2953 tablet_service.cc:1467] Processing CreateTablet for tablet 78bc21a85343470099406b858ea13583 (DEFAULT_TABLE table=test-table [id=7093241c62ec405aae69104d59a9fc8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:58.812863 2814 tablet_service.cc:1467] Processing CreateTablet for tablet 78bc21a85343470099406b858ea13583 (DEFAULT_TABLE table=test-table [id=7093241c62ec405aae69104d59a9fc8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:58.812984 2953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bc21a85343470099406b858ea13583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:58.812984 2674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bc21a85343470099406b858ea13583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:58.813017 2814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bc21a85343470099406b858ea13583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:58.814409 3133 tablet_bootstrap.cc:492] T 78bc21a85343470099406b858ea13583 P 068176828f894b1599b47b45654b36c8: Bootstrap starting.
I20260620 19:07:58.814430 3132 tablet_bootstrap.cc:492] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516: Bootstrap starting.
I20260620 19:07:58.814812 3134 tablet_bootstrap.cc:492] T 78bc21a85343470099406b858ea13583 P 03fee9c5de834f9695a7976cb0503ca3: Bootstrap starting.
I20260620 19:07:58.815162 3133 tablet_bootstrap.cc:654] T 78bc21a85343470099406b858ea13583 P 068176828f894b1599b47b45654b36c8: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:58.815181 3132 tablet_bootstrap.cc:654] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:58.815270 3134 tablet_bootstrap.cc:654] T 78bc21a85343470099406b858ea13583 P 03fee9c5de834f9695a7976cb0503ca3: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:58.815863 3132 tablet_bootstrap.cc:492] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516: No bootstrap required, opened a new log
I20260620 19:07:58.815923 3132 ts_tablet_manager.cc:1397] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:07:58.815933 3134 tablet_bootstrap.cc:492] T 78bc21a85343470099406b858ea13583 P 03fee9c5de834f9695a7976cb0503ca3: No bootstrap required, opened a new log
I20260620 19:07:58.815984 3134 ts_tablet_manager.cc:1397] T 78bc21a85343470099406b858ea13583 P 03fee9c5de834f9695a7976cb0503ca3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:07:58.816073 3132 raft_consensus.cc:348] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:58.816176 3132 raft_consensus.cc:374] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:58.816210 3132 raft_consensus.cc:729] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f315f8c5ed9f4fe38f59de5c7f238516, State: Initialized, Role: FOLLOWER
I20260620 19:07:58.816172 3134 raft_consensus.cc:348] T 78bc21a85343470099406b858ea13583 P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:58.816262 3134 raft_consensus.cc:374] T 78bc21a85343470099406b858ea13583 P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:58.816284 3134 raft_consensus.cc:729] T 78bc21a85343470099406b858ea13583 P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03fee9c5de834f9695a7976cb0503ca3, State: Initialized, Role: FOLLOWER
I20260620 19:07:58.816267 3132 consensus_queue.cc:260] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:58.816342 3134 consensus_queue.cc:260] T 78bc21a85343470099406b858ea13583 P 03fee9c5de834f9695a7976cb0503ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:58.816448 3134 ts_tablet_manager.cc:1428] T 78bc21a85343470099406b858ea13583 P 03fee9c5de834f9695a7976cb0503ca3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:58.816455 3132 ts_tablet_manager.cc:1428] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 19:07:58.816668 3133 tablet_bootstrap.cc:492] T 78bc21a85343470099406b858ea13583 P 068176828f894b1599b47b45654b36c8: No bootstrap required, opened a new log
I20260620 19:07:58.816725 3133 ts_tablet_manager.cc:1397] T 78bc21a85343470099406b858ea13583 P 068176828f894b1599b47b45654b36c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:07:58.816851 3133 raft_consensus.cc:348] T 78bc21a85343470099406b858ea13583 P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:58.816939 3133 raft_consensus.cc:374] T 78bc21a85343470099406b858ea13583 P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:58.816960 3133 raft_consensus.cc:729] T 78bc21a85343470099406b858ea13583 P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 068176828f894b1599b47b45654b36c8, State: Initialized, Role: FOLLOWER
I20260620 19:07:58.817020 3133 consensus_queue.cc:260] T 78bc21a85343470099406b858ea13583 P 068176828f894b1599b47b45654b36c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:58.817158 3133 ts_tablet_manager.cc:1428] T 78bc21a85343470099406b858ea13583 P 068176828f894b1599b47b45654b36c8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:59.016094 3138 raft_consensus.cc:482] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:07:59.016214 3138 raft_consensus.cc:504] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:59.016623 3138 leader_election.cc:288] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03fee9c5de834f9695a7976cb0503ca3 (127.0.203.1:40833), 068176828f894b1599b47b45654b36c8 (127.0.203.3:41499)
I20260620 19:07:59.020159 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:59.016815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:40833 (local address 127.0.203.2:42495)
0620 19:07:59.017011 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:59.017056 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:59.017125 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:59.017309 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:59.017310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:59.017312 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:59.017415 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:59.017417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:59.018140 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:59.018141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:59.019177 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:59.019180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:59.019845 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:59.019846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:59.019932 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:59.019936 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:07:59.019999 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 19:07:59.020070 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":22,"thread_start_us":126,"threads_started":1}
I20260620 19:07:59.020357 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:59.016843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:41499 (local address 127.0.203.2:41393)
0620 19:07:59.017020 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:59.017032 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:59.017091 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:59.017255 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:59.017256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:59.017259 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:07:59.017345 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:59.017347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:59.018101 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:59.018102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:59.019461 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:59.019464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:59.020077 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:59.020078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:59.020149 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:59.020154 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:07:59.020216 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:07:59.020260 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":50,"thread_start_us":108,"threads_started":1}
I20260620 19:07:59.020375 3142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:59.016908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:41393 (local address 127.0.203.3:41499)
0620 19:07:59.017092 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:59.017095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:59.017126 (+ 31us) server_negotiation.cc:407] Connection header received
0620 19:07:59.017157 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:59.017158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:59.017169 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:59.017238 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:59.017421 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:59.018020 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:59.019550 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:59.020001 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:59.020064 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:59.020075 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:59.020241 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 19:07:59.020289 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":73,"threads_started":1}
I20260620 19:07:59.020179 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:59.016864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:42495 (local address 127.0.203.1:40833)
0620 19:07:59.017059 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:59.017061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:59.017111 (+ 50us) server_negotiation.cc:407] Connection header received
0620 19:07:59.017188 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:59.017188 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:59.017198 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:59.017309 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:07:59.017487 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:59.018038 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:59.019271 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:59.019769 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:59.019823 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:59.019835 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:59.020012 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 19:07:59.020070 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":93,"threads_started":1}
I20260620 19:07:59.020650 2974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bc21a85343470099406b858ea13583" candidate_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "068176828f894b1599b47b45654b36c8" is_pre_election: true
I20260620 19:07:59.020745 2974 raft_consensus.cc:2454] T 78bc21a85343470099406b858ea13583 P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f315f8c5ed9f4fe38f59de5c7f238516 in term 0.
I20260620 19:07:59.020758 2694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bc21a85343470099406b858ea13583" candidate_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03fee9c5de834f9695a7976cb0503ca3" is_pre_election: true
I20260620 19:07:59.020861 2694 raft_consensus.cc:2454] T 78bc21a85343470099406b858ea13583 P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f315f8c5ed9f4fe38f59de5c7f238516 in term 0.
I20260620 19:07:59.020921 2767 leader_election.cc:302] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 068176828f894b1599b47b45654b36c8, f315f8c5ed9f4fe38f59de5c7f238516; no voters:
I20260620 19:07:59.021111 3138 raft_consensus.cc:2781] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:07:59.021170 3138 raft_consensus.cc:482] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:07:59.021203 3138 raft_consensus.cc:3037] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:59.042793 3138 raft_consensus.cc:504] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:59.043045 3138 leader_election.cc:288] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [CANDIDATE]: Term 1 election: Requested vote from peers 03fee9c5de834f9695a7976cb0503ca3 (127.0.203.1:40833), 068176828f894b1599b47b45654b36c8 (127.0.203.3:41499)
I20260620 19:07:59.043300 2694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bc21a85343470099406b858ea13583" candidate_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03fee9c5de834f9695a7976cb0503ca3"
I20260620 19:07:59.043390 2694 raft_consensus.cc:3037] T 78bc21a85343470099406b858ea13583 P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:59.043304 2974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bc21a85343470099406b858ea13583" candidate_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "068176828f894b1599b47b45654b36c8"
I20260620 19:07:59.043432 2974 raft_consensus.cc:3037] T 78bc21a85343470099406b858ea13583 P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:59.044035 2694 raft_consensus.cc:2454] T 78bc21a85343470099406b858ea13583 P 03fee9c5de834f9695a7976cb0503ca3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f315f8c5ed9f4fe38f59de5c7f238516 in term 1.
I20260620 19:07:59.044287 2768 leader_election.cc:302] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03fee9c5de834f9695a7976cb0503ca3, f315f8c5ed9f4fe38f59de5c7f238516; no voters:
I20260620 19:07:59.044358 2974 raft_consensus.cc:2454] T 78bc21a85343470099406b858ea13583 P 068176828f894b1599b47b45654b36c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f315f8c5ed9f4fe38f59de5c7f238516 in term 1.
I20260620 19:07:59.044394 3138 raft_consensus.cc:2781] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:07:59.044564 3138 raft_consensus.cc:686] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [term 1 LEADER]: Becoming Leader. State: Replica: f315f8c5ed9f4fe38f59de5c7f238516, State: Running, Role: LEADER
I20260620 19:07:59.044641 3138 consensus_queue.cc:237] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:59.045480 2537 catalog_manager.cc:5495] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 reported cstate change: term changed from 0 to 1, leader changed from <none> to f315f8c5ed9f4fe38f59de5c7f238516 (127.0.203.2). New cstate: current_term: 1 leader_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } health_report { overall_health: UNKNOWN } } }
I20260620 19:07:59.103158 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:59.100349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45038 (local address 127.0.203.2:45187)
0620 19:07:59.100509 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:59.100511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:59.100518 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:07:59.100566 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:59.100566 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:59.100585 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:59.100700 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:07:59.100850 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:59.101364 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:59.102171 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:59.102454 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:59.102516 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:59.102719 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:07:59.102890 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:07:59.102959 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:59.103007 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 19:07:59.103030 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":80,"threads_started":1}
I20260620 19:07:59.108881 3142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:59.106490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:34755 (local address 127.0.203.3:41499)
0620 19:07:59.106563 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:59.106565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:59.106672 (+ 107us) server_negotiation.cc:407] Connection header received
0620 19:07:59.106702 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:59.106703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:59.106715 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:59.106778 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:07:59.106934 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:59.107423 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:59.108263 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:59.108450 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:59.108518 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:59.108665 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:07:59.108692 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:07:59.108730 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:59.108796 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 19:07:59.108816 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:07:59.108893 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:59.106371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:41499 (local address 127.0.203.2:34755)
0620 19:07:59.106591 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:59.106609 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:59.106660 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:59.106790 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:59.106791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:59.106793 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:07:59.106848 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:59.106849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:59.107520 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:59.107521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:59.108165 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:59.108167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:59.108521 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:59.108522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:59.108575 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:59.108589 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:07:59.108744 (+ 155us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:07:59.108746 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:07:59.108781 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 19:07:59.108821 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":85,"threads_started":1}
I20260620 19:07:59.111294 2974 raft_consensus.cc:1261] T 78bc21a85343470099406b858ea13583 P 068176828f894b1599b47b45654b36c8 [term 1 FOLLOWER]: Refusing update from remote peer f315f8c5ed9f4fe38f59de5c7f238516: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:59.111342 2694 raft_consensus.cc:1261] T 78bc21a85343470099406b858ea13583 P 03fee9c5de834f9695a7976cb0503ca3 [term 1 FOLLOWER]: Refusing update from remote peer f315f8c5ed9f4fe38f59de5c7f238516: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:59.111580 3138 consensus_queue.cc:1035] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:59.111742 3143 consensus_queue.cc:1035] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:59.119557 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:59.116877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:45501 (local address 127.0.203.2:45187)
0620 19:07:59.116923 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:07:59.116925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:07:59.117046 (+ 121us) server_negotiation.cc:407] Connection header received
0620 19:07:59.117105 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:07:59.117106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:07:59.117118 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:07:59.117170 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:07:59.117309 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:59.117786 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:59.118634 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:07:59.118904 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:07:59.119071 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:59.119274 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:07:59.119316 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:07:59.119386 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:07:59.119449 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 19:07:59.119473 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:07:59.119575 3154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:07:59.116843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:45187 (local address 127.0.203.3:45501)
0620 19:07:59.116971 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:07:59.116982 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:07:59.117044 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:07:59.117179 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:07:59.117180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:07:59.117182 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:07:59.117237 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:59.117239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:59.117862 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:59.117863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:59.118540 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:07:59.118542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:07:59.118990 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:07:59.118991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:07:59.119071 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:07:59.119077 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:07:59.119388 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:07:59.119390 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:07:59.119444 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 19:07:59.119497 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":83,"threads_started":1}
I20260620 19:07:59.130244 2536 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:07:59.130379 2536 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:07:59.131064 2536 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b: Sending DeleteTablet for 3 replicas of tablet 78bc21a85343470099406b858ea13583
I20260620 19:07:59.131366 2814 tablet_service.cc:1514] Processing DeleteTablet for tablet 78bc21a85343470099406b858ea13583 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:59 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:45036
I20260620 19:07:59.131374 2674 tablet_service.cc:1514] Processing DeleteTablet for tablet 78bc21a85343470099406b858ea13583 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:59 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:35218
I20260620 19:07:59.131412 2953 tablet_service.cc:1514] Processing DeleteTablet for tablet 78bc21a85343470099406b858ea13583 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:07:59 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:43256
I20260620 19:07:59.131506 3124 tablet_replica.cc:330] T 78bc21a85343470099406b858ea13583 P 068176828f894b1599b47b45654b36c8: stopping tablet replica
I20260620 19:07:59.131487 3121 tablet_replica.cc:330] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516: stopping tablet replica
I20260620 19:07:59.131558 3121 raft_consensus.cc:2229] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:07:59.131560 3124 raft_consensus.cc:2229] T 78bc21a85343470099406b858ea13583 P 068176828f894b1599b47b45654b36c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:07:59.131606 3124 raft_consensus.cc:2258] T 78bc21a85343470099406b858ea13583 P 068176828f894b1599b47b45654b36c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:59.131644 3121 raft_consensus.cc:2258] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:59.131744 3122 tablet_replica.cc:330] T 78bc21a85343470099406b858ea13583 P 03fee9c5de834f9695a7976cb0503ca3: stopping tablet replica
I20260620 19:07:59.131821 3122 raft_consensus.cc:2229] T 78bc21a85343470099406b858ea13583 P 03fee9c5de834f9695a7976cb0503ca3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:07:59.131860 3122 raft_consensus.cc:2258] T 78bc21a85343470099406b858ea13583 P 03fee9c5de834f9695a7976cb0503ca3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:07:59.131958 3124 ts_tablet_manager.cc:1905] T 78bc21a85343470099406b858ea13583 P 068176828f894b1599b47b45654b36c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:59.131958 3121 ts_tablet_manager.cc:1905] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:59.132180 3122 ts_tablet_manager.cc:1905] T 78bc21a85343470099406b858ea13583 P 03fee9c5de834f9695a7976cb0503ca3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:07:59.133123 2536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:07:59.133275 2536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:07:59.134778 2953 tablet_service.cc:1467] Processing CreateTablet for tablet ebfd5ad0301f44138095dedb13884ec6 (DEFAULT_TABLE table=test-table [id=01d3a58765e840e1ab892cf5e6d1a760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:59.134786 2814 tablet_service.cc:1467] Processing CreateTablet for tablet ebfd5ad0301f44138095dedb13884ec6 (DEFAULT_TABLE table=test-table [id=01d3a58765e840e1ab892cf5e6d1a760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:59.134780 2674 tablet_service.cc:1467] Processing CreateTablet for tablet ebfd5ad0301f44138095dedb13884ec6 (DEFAULT_TABLE table=test-table [id=01d3a58765e840e1ab892cf5e6d1a760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:07:59.141021 2953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebfd5ad0301f44138095dedb13884ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:59.141194 2674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebfd5ad0301f44138095dedb13884ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:59.142522 3134 tablet_bootstrap.cc:492] T ebfd5ad0301f44138095dedb13884ec6 P 03fee9c5de834f9695a7976cb0503ca3: Bootstrap starting.
I20260620 19:07:59.142745 3133 tablet_bootstrap.cc:492] T ebfd5ad0301f44138095dedb13884ec6 P 068176828f894b1599b47b45654b36c8: Bootstrap starting.
I20260620 19:07:59.142884 3134 tablet_bootstrap.cc:654] T ebfd5ad0301f44138095dedb13884ec6 P 03fee9c5de834f9695a7976cb0503ca3: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:59.143077 3133 tablet_bootstrap.cc:654] T ebfd5ad0301f44138095dedb13884ec6 P 068176828f894b1599b47b45654b36c8: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:59.143896 3133 tablet_bootstrap.cc:492] T ebfd5ad0301f44138095dedb13884ec6 P 068176828f894b1599b47b45654b36c8: No bootstrap required, opened a new log
I20260620 19:07:59.143951 3133 ts_tablet_manager.cc:1397] T ebfd5ad0301f44138095dedb13884ec6 P 068176828f894b1599b47b45654b36c8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:07:59.144065 3133 raft_consensus.cc:348] T ebfd5ad0301f44138095dedb13884ec6 P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:59.144133 3133 raft_consensus.cc:374] T ebfd5ad0301f44138095dedb13884ec6 P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:59.144155 3133 raft_consensus.cc:729] T ebfd5ad0301f44138095dedb13884ec6 P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 068176828f894b1599b47b45654b36c8, State: Initialized, Role: FOLLOWER
I20260620 19:07:59.144204 3133 consensus_queue.cc:260] T ebfd5ad0301f44138095dedb13884ec6 P 068176828f894b1599b47b45654b36c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:59.144304 3133 ts_tablet_manager.cc:1428] T ebfd5ad0301f44138095dedb13884ec6 P 068176828f894b1599b47b45654b36c8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:07:59.144678 3134 tablet_bootstrap.cc:492] T ebfd5ad0301f44138095dedb13884ec6 P 03fee9c5de834f9695a7976cb0503ca3: No bootstrap required, opened a new log
I20260620 19:07:59.144755 3134 ts_tablet_manager.cc:1397] T ebfd5ad0301f44138095dedb13884ec6 P 03fee9c5de834f9695a7976cb0503ca3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:07:59.144884 3134 raft_consensus.cc:348] T ebfd5ad0301f44138095dedb13884ec6 P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:59.144958 3134 raft_consensus.cc:374] T ebfd5ad0301f44138095dedb13884ec6 P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:59.144990 3134 raft_consensus.cc:729] T ebfd5ad0301f44138095dedb13884ec6 P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03fee9c5de834f9695a7976cb0503ca3, State: Initialized, Role: FOLLOWER
I20260620 19:07:59.145047 3134 consensus_queue.cc:260] T ebfd5ad0301f44138095dedb13884ec6 P 03fee9c5de834f9695a7976cb0503ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:59.145147 3134 ts_tablet_manager.cc:1428] T ebfd5ad0301f44138095dedb13884ec6 P 03fee9c5de834f9695a7976cb0503ca3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 19:07:59.145328 2814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebfd5ad0301f44138095dedb13884ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:07:59.146356 3124 ts_tablet_manager.cc:1918] T 78bc21a85343470099406b858ea13583 P 068176828f894b1599b47b45654b36c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:07:59.146426 3124 log.cc:1196] T 78bc21a85343470099406b858ea13583 P 068176828f894b1599b47b45654b36c8: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/78bc21a85343470099406b858ea13583
I20260620 19:07:59.146638 3132 tablet_bootstrap.cc:492] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516: Bootstrap starting.
I20260620 19:07:59.146664 3124 ts_tablet_manager.cc:1939] T 78bc21a85343470099406b858ea13583 P 068176828f894b1599b47b45654b36c8: Deleting consensus metadata
I20260620 19:07:59.147045 3132 tablet_bootstrap.cc:654] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516: Neither blocks nor log segments found. Creating new log.
I20260620 19:07:59.147799 3132 tablet_bootstrap.cc:492] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516: No bootstrap required, opened a new log
I20260620 19:07:59.147859 3132 ts_tablet_manager.cc:1397] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:07:59.148036 3132 raft_consensus.cc:348] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:59.148113 3132 raft_consensus.cc:374] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:07:59.148132 3132 raft_consensus.cc:729] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f315f8c5ed9f4fe38f59de5c7f238516, State: Initialized, Role: FOLLOWER
I20260620 19:07:59.148185 3132 consensus_queue.cc:260] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:59.148279 3132 ts_tablet_manager.cc:1428] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:07:59.153304 3122 ts_tablet_manager.cc:1918] T 78bc21a85343470099406b858ea13583 P 03fee9c5de834f9695a7976cb0503ca3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:07:59.153395 3122 log.cc:1196] T 78bc21a85343470099406b858ea13583 P 03fee9c5de834f9695a7976cb0503ca3: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-0/wal/wals/78bc21a85343470099406b858ea13583
I20260620 19:07:59.153641 3122 ts_tablet_manager.cc:1939] T 78bc21a85343470099406b858ea13583 P 03fee9c5de834f9695a7976cb0503ca3: Deleting consensus metadata
I20260620 19:07:59.188163 2524 catalog_manager.cc:4841] TS 03fee9c5de834f9695a7976cb0503ca3 (127.0.203.1:40833): tablet 78bc21a85343470099406b858ea13583 (table test-table [id=7093241c62ec405aae69104d59a9fc8f]) successfully deleted
I20260620 19:07:59.221800 2523 catalog_manager.cc:4841] TS 068176828f894b1599b47b45654b36c8 (127.0.203.3:41499): tablet 78bc21a85343470099406b858ea13583 (table test-table [id=7093241c62ec405aae69104d59a9fc8f]) successfully deleted
I20260620 19:07:59.235793 3121 ts_tablet_manager.cc:1918] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:07:59.235904 3121 log.cc:1196] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/78bc21a85343470099406b858ea13583
I20260620 19:07:59.236143 3121 ts_tablet_manager.cc:1939] T 78bc21a85343470099406b858ea13583 P f315f8c5ed9f4fe38f59de5c7f238516: Deleting consensus metadata
I20260620 19:07:59.241262 2524 catalog_manager.cc:4841] TS f315f8c5ed9f4fe38f59de5c7f238516 (127.0.203.2:45187): tablet 78bc21a85343470099406b858ea13583 (table test-table [id=7093241c62ec405aae69104d59a9fc8f]) successfully deleted
I20260620 19:07:59.400054 3143 raft_consensus.cc:482] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:07:59.400161 3143 raft_consensus.cc:504] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:59.400465 3143 leader_election.cc:288] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03fee9c5de834f9695a7976cb0503ca3 (127.0.203.1:40833), 068176828f894b1599b47b45654b36c8 (127.0.203.3:41499)
I20260620 19:07:59.400998 2694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebfd5ad0301f44138095dedb13884ec6" candidate_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03fee9c5de834f9695a7976cb0503ca3" is_pre_election: true
I20260620 19:07:59.401106 2694 raft_consensus.cc:2454] T ebfd5ad0301f44138095dedb13884ec6 P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f315f8c5ed9f4fe38f59de5c7f238516 in term 0.
I20260620 19:07:59.401017 2974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebfd5ad0301f44138095dedb13884ec6" candidate_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "068176828f894b1599b47b45654b36c8" is_pre_election: true
I20260620 19:07:59.401190 2974 raft_consensus.cc:2454] T ebfd5ad0301f44138095dedb13884ec6 P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f315f8c5ed9f4fe38f59de5c7f238516 in term 0.
I20260620 19:07:59.401357 2768 leader_election.cc:302] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03fee9c5de834f9695a7976cb0503ca3, f315f8c5ed9f4fe38f59de5c7f238516; no voters:
I20260620 19:07:59.401481 3143 raft_consensus.cc:2781] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:07:59.401535 3143 raft_consensus.cc:482] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:07:59.401568 3143 raft_consensus.cc:3037] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:59.434665 3143 raft_consensus.cc:504] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:59.435053 3143 leader_election.cc:288] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [CANDIDATE]: Term 1 election: Requested vote from peers 03fee9c5de834f9695a7976cb0503ca3 (127.0.203.1:40833), 068176828f894b1599b47b45654b36c8 (127.0.203.3:41499)
I20260620 19:07:59.435302 2974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebfd5ad0301f44138095dedb13884ec6" candidate_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "068176828f894b1599b47b45654b36c8"
I20260620 19:07:59.435299 2694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebfd5ad0301f44138095dedb13884ec6" candidate_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03fee9c5de834f9695a7976cb0503ca3"
I20260620 19:07:59.435395 2974 raft_consensus.cc:3037] T ebfd5ad0301f44138095dedb13884ec6 P 068176828f894b1599b47b45654b36c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:59.435396 2694 raft_consensus.cc:3037] T ebfd5ad0301f44138095dedb13884ec6 P 03fee9c5de834f9695a7976cb0503ca3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:07:59.436256 2974 raft_consensus.cc:2454] T ebfd5ad0301f44138095dedb13884ec6 P 068176828f894b1599b47b45654b36c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f315f8c5ed9f4fe38f59de5c7f238516 in term 1.
I20260620 19:07:59.436256 2694 raft_consensus.cc:2454] T ebfd5ad0301f44138095dedb13884ec6 P 03fee9c5de834f9695a7976cb0503ca3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f315f8c5ed9f4fe38f59de5c7f238516 in term 1.
I20260620 19:07:59.436501 2768 leader_election.cc:302] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03fee9c5de834f9695a7976cb0503ca3, f315f8c5ed9f4fe38f59de5c7f238516; no voters:
I20260620 19:07:59.436635 3143 raft_consensus.cc:2781] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:07:59.436691 3143 raft_consensus.cc:686] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [term 1 LEADER]: Becoming Leader. State: Replica: f315f8c5ed9f4fe38f59de5c7f238516, State: Running, Role: LEADER
I20260620 19:07:59.436748 3143 consensus_queue.cc:237] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } }
I20260620 19:07:59.437448 2536 catalog_manager.cc:5495] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 reported cstate change: term changed from 0 to 1, leader changed from <none> to f315f8c5ed9f4fe38f59de5c7f238516 (127.0.203.2). New cstate: current_term: 1 leader_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f315f8c5ed9f4fe38f59de5c7f238516" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 } health_report { overall_health: UNKNOWN } } }
I20260620 19:07:59.497546 2694 raft_consensus.cc:1261] T ebfd5ad0301f44138095dedb13884ec6 P 03fee9c5de834f9695a7976cb0503ca3 [term 1 FOLLOWER]: Refusing update from remote peer f315f8c5ed9f4fe38f59de5c7f238516: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:59.497572 2974 raft_consensus.cc:1261] T ebfd5ad0301f44138095dedb13884ec6 P 068176828f894b1599b47b45654b36c8 [term 1 FOLLOWER]: Refusing update from remote peer f315f8c5ed9f4fe38f59de5c7f238516: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:07:59.497831 3152 consensus_queue.cc:1035] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03fee9c5de834f9695a7976cb0503ca3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:07:59.497979 3143 consensus_queue.cc:1035] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "068176828f894b1599b47b45654b36c8" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:07:59.502732 3104 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 19:08:01.026067 2537 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:01.026176 2537 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:01.027182 2537 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e467ed4e966a48b193c5df2a00b50f8b: Sending DeleteTablet for 3 replicas of tablet ebfd5ad0301f44138095dedb13884ec6
I20260620 19:08:01.027503 2674 tablet_service.cc:1514] Processing DeleteTablet for tablet ebfd5ad0301f44138095dedb13884ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:01 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:35218
I20260620 19:08:01.027501 2953 tablet_service.cc:1514] Processing DeleteTablet for tablet ebfd5ad0301f44138095dedb13884ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:01 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:43256
I20260620 19:08:01.027508 2814 tablet_service.cc:1514] Processing DeleteTablet for tablet ebfd5ad0301f44138095dedb13884ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:01 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:45036
I20260620 19:08:01.027750 3185 tablet_replica.cc:330] T ebfd5ad0301f44138095dedb13884ec6 P 068176828f894b1599b47b45654b36c8: stopping tablet replica
I20260620 19:08:01.027750 3186 tablet_replica.cc:330] T ebfd5ad0301f44138095dedb13884ec6 P 03fee9c5de834f9695a7976cb0503ca3: stopping tablet replica
I20260620 19:08:01.027817 3185 raft_consensus.cc:2229] T ebfd5ad0301f44138095dedb13884ec6 P 068176828f894b1599b47b45654b36c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:01.027835 3186 raft_consensus.cc:2229] T ebfd5ad0301f44138095dedb13884ec6 P 03fee9c5de834f9695a7976cb0503ca3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:01.027869 3185 raft_consensus.cc:2258] T ebfd5ad0301f44138095dedb13884ec6 P 068176828f894b1599b47b45654b36c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:01.027876 3186 raft_consensus.cc:2258] T ebfd5ad0301f44138095dedb13884ec6 P 03fee9c5de834f9695a7976cb0503ca3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:01.027861 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 2609
I20260620 19:08:01.028620 3185 ts_tablet_manager.cc:1905] T ebfd5ad0301f44138095dedb13884ec6 P 068176828f894b1599b47b45654b36c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:01.029651 3184 tablet_replica.cc:330] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516: stopping tablet replica
I20260620 19:08:01.029749 3184 raft_consensus.cc:2229] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:08:01.029840 3184 raft_consensus.cc:2258] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:01.030095 3184 ts_tablet_manager.cc:1905] T ebfd5ad0301f44138095dedb13884ec6 P f315f8c5ed9f4fe38f59de5c7f238516: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:01.034891 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 2749
W20260620 19:08:01.035923 2524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.1:40833 (error 108)
I20260620 19:08:01.040299 3188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:01.038038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:40833 (local address 127.0.0.1:35230)
0620 19:08:01.040188 (+ 2150us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:01.040250 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.1:40833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2118,"thread_start_us":86,"threads_started":1}
W20260620 19:08:01.040495 2524 catalog_manager.cc:4584] TS 03fee9c5de834f9695a7976cb0503ca3 (127.0.203.1:40833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ebfd5ad0301f44138095dedb13884ec6: Network error: Client connection negotiation failed: client connection to 127.0.203.1:40833: connect: Connection refused (error 111)
I20260620 19:08:01.042019 3188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:01.041899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:45187 (local address 127.0.0.1:45050)
0620 19:08:01.041955 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:01.041989 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.2:45187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 19:08:01.042028 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 2889
I20260620 19:08:01.057680 3188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:01.057523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:41499 (local address 127.0.0.1:43270)
0620 19:08:01.057587 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:01.057622 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.3:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 19:08:01.057698 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 2505
2026-06-20T19:08:01Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6316 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:08:01 dist-test-slave-wd30 krb5kdc[3191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:08:01 dist-test-slave-wd30 krb5kdc[3191](info): set up 2 sockets
Jun 20 19:08:01 dist-test-slave-wd30 krb5kdc[3191](info): commencing operation
krb5kdc: starting...
W20260620 19:08:02.451438 812 mini_kdc.cc:121] Time spent starting KDC: real 1.371s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:08:02 dist-test-slave-wd30 krb5kdc[3191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:08:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:08:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:08:02.571997 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:39645
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:39645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:02.656670 3204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:02.656903 3204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:02.656927 3204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:02.658195 3204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:02.658242 3204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:02.658257 3204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:02.658272 3204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:02.658284 3204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:08:02.660037 3204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:39645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:39645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:02.660305 3204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:02.660559 3204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:02.665582 3210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:02.665584 3212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:02.665804 3204 server_base.cc:1034] running on GCE node
W20260620 19:08:02.665625 3209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:02.666141 3204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:02.666456 3204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:02.667632 3204 hybrid_clock.cc:648] HybridClock initialized: now 1781982482667615 us; error 29 us; skew 500 ppm
Jun 20 19:08:02 dist-test-slave-wd30 krb5kdc[3191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:02.670004 3204 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:08:02.670430 3204 webserver.cc:460] Webserver started at http://127.0.203.62:36563/ using document root <none> and password file <none>
I20260620 19:08:02.670673 3204 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:02.670744 3204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:02.670889 3204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:02.672021 3204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "16023ca812724dbca3980b0bd6e64b33"
format_stamp: "Formatted at 2026-06-20 19:08:02 on dist-test-slave-wd30"
server_key: "c73873642023658eaa42571470eac031"
server_key_iv: "cd1f7da4e2ef13a39083078e61bf9406"
server_key_version: "encryptionkey@0"
I20260620 19:08:02.672375 3204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "16023ca812724dbca3980b0bd6e64b33"
format_stamp: "Formatted at 2026-06-20 19:08:02 on dist-test-slave-wd30"
server_key: "c73873642023658eaa42571470eac031"
server_key_iv: "cd1f7da4e2ef13a39083078e61bf9406"
server_key_version: "encryptionkey@0"
I20260620 19:08:02.673655 3204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:08:02.674342 3219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:02.674507 3204 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:02.674605 3204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "16023ca812724dbca3980b0bd6e64b33"
format_stamp: "Formatted at 2026-06-20 19:08:02 on dist-test-slave-wd30"
server_key: "c73873642023658eaa42571470eac031"
server_key_iv: "cd1f7da4e2ef13a39083078e61bf9406"
server_key_version: "encryptionkey@0"
I20260620 19:08:02.674686 3204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:02.696465 3204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:02.698686 3204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:02.698880 3204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:02.703063 3204 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:39645
I20260620 19:08:02.703373 3271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:39645 every 8 connection(s)
I20260620 19:08:02.703576 3204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:02.704319 3272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:02.706811 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 3204
I20260620 19:08:02.706808 3272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33: Bootstrap starting.
I20260620 19:08:02.706923 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:08:02.707094 812 external_mini_cluster.cc:1428] Setting key ed12594e0a094fa480687d3e5ac0ea1b
I20260620 19:08:02.707677 3272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:02.708032 3272 log.cc:824] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:02.708869 3272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33: No bootstrap required, opened a new log
I20260620 19:08:02.711201 3272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16023ca812724dbca3980b0bd6e64b33" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39645 } }
I20260620 19:08:02.711390 3272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:02.711422 3272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16023ca812724dbca3980b0bd6e64b33, State: Initialized, Role: FOLLOWER
I20260620 19:08:02.711567 3272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16023ca812724dbca3980b0bd6e64b33" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39645 } }
I20260620 19:08:02.711676 3272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:02.711714 3272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:02.711767 3272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:02.712484 3272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16023ca812724dbca3980b0bd6e64b33" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39645 } }
I20260620 19:08:02.712610 3272 leader_election.cc:302] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16023ca812724dbca3980b0bd6e64b33; no voters:
I20260620 19:08:02.712792 3272 leader_election.cc:288] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:08:02.712823 3277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:02.712949 3277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33 [term 1 LEADER]: Becoming Leader. State: Replica: 16023ca812724dbca3980b0bd6e64b33, State: Running, Role: LEADER
I20260620 19:08:02.713092 3272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:02.713073 3277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16023ca812724dbca3980b0bd6e64b33" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39645 } }
I20260620 19:08:02.713418 3277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16023ca812724dbca3980b0bd6e64b33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16023ca812724dbca3980b0bd6e64b33" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39645 } } }
I20260620 19:08:02.713577 3277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33 [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:02.713778 3277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16023ca812724dbca3980b0bd6e64b33. Latest consensus state: current_term: 1 leader_uuid: "16023ca812724dbca3980b0bd6e64b33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16023ca812724dbca3980b0bd6e64b33" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39645 } } }
I20260620 19:08:02.713908 3277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33 [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:02.714296 3284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 19:08:02 dist-test-slave-wd30 krb5kdc[3191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:02.714915 3284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:08:02.716902 3284 catalog_manager.cc:1347] Generated new cluster ID: fc46067750dd4eb9ab788a8dd5306a6a
I20260620 19:08:02.716962 3284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:08:02.719506 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:02.707752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36194 (local address 127.0.203.62:39645)
0620 19:08:02.707937 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:02.707948 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:02.707983 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:08:02.708493 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:02.708501 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:02.708554 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:02.708686 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:02.709529 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:02.710445 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:02.711662 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:02.712199 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:02.712289 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:02.715481 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:02.715485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:02.715486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:02.715499 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:02.717743 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:02.718192 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:02.718195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:02.718196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:02.718303 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:02.718528 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:02.718529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:02.718529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:02.718832 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:02.718992 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:02.719148 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 19:08:02.719283 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260620 19:08:02.731040 3284 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:08:02.731652 3284 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:02.738232 3284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16023ca812724dbca3980b0bd6e64b33: Generated new TSK 0
I20260620 19:08:02.738515 3284 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
I20260620 19:08:02.768280 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:39645
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:02.867470 3298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:02.867698 3298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:02.867738 3298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:02.868917 3298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:02.868983 3298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:02.869040 3298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:08:02.870556 3298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:02.870831 3298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:02.871059 3298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:02.876091 3306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:02.876134 3298 server_base.cc:1034] running on GCE node
W20260620 19:08:02.876091 3303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:02.876094 3304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:02.876582 3298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:02.876874 3298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:02.878027 3298 hybrid_clock.cc:648] HybridClock initialized: now 1781982482878005 us; error 38 us; skew 500 ppm
Jun 20 19:08:02 dist-test-slave-wd30 krb5kdc[3191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:02.880569 3298 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:08:02.880975 3298 webserver.cc:460] Webserver started at http://127.0.203.1:43159/ using document root <none> and password file <none>
I20260620 19:08:02.881210 3298 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:02.881281 3298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:02.881433 3298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:02.882467 3298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "4579f9d93d2f4ec7afdec3c47c3ff720"
format_stamp: "Formatted at 2026-06-20 19:08:02 on dist-test-slave-wd30"
server_key: "03e8230504e8a8b68180b24c99b10181"
server_key_iv: "ad631407d09f625543f55006210b1273"
server_key_version: "encryptionkey@0"
I20260620 19:08:02.882894 3298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "4579f9d93d2f4ec7afdec3c47c3ff720"
format_stamp: "Formatted at 2026-06-20 19:08:02 on dist-test-slave-wd30"
server_key: "03e8230504e8a8b68180b24c99b10181"
server_key_iv: "ad631407d09f625543f55006210b1273"
server_key_version: "encryptionkey@0"
I20260620 19:08:02.884227 3298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:08:02.884913 3313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:02.885097 3298 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:02.885197 3298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "4579f9d93d2f4ec7afdec3c47c3ff720"
format_stamp: "Formatted at 2026-06-20 19:08:02 on dist-test-slave-wd30"
server_key: "03e8230504e8a8b68180b24c99b10181"
server_key_iv: "ad631407d09f625543f55006210b1273"
server_key_version: "encryptionkey@0"
I20260620 19:08:02.885278 3298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:02.895820 3298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:02.897902 3298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:02.898079 3298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:02.898340 3298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:02.898670 3298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:02.898730 3298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:02.898772 3298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:02.898814 3298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:02.904865 3298 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:44513
I20260620 19:08:02.904937 3426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:44513 every 8 connection(s)
I20260620 19:08:02.905299 3298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:08:02 dist-test-slave-wd30 krb5kdc[3191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:02.913990 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 3298
I20260620 19:08:02.914095 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:08:02.914304 812 external_mini_cluster.cc:1428] Setting key 29c2092f2ec2829cabaa9866b39b2bab
I20260620 19:08:02.915874 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:02.906102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:38641 (local address 127.0.203.62:39645)
0620 19:08:02.906171 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:02.906173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:02.906392 (+ 219us) server_negotiation.cc:407] Connection header received
0620 19:08:02.907529 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:02.907530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:02.907572 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:02.907625 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:02.908307 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:02.908846 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:02.909895 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:02.910117 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:02.910203 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:02.912519 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:02.912521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:02.912521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:02.912528 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:02.914010 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:02.914573 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:02.914575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:02.914576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:02.914654 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:02.915025 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:02.915026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:02.915026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:02.915256 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:02.915352 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:02.915617 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 19:08:02.915726 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:08:02.915895 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:02.906075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:39645 (local address 127.0.203.1:38641)
0620 19:08:02.906265 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:02.906305 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:02.907415 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:02.907713 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:02.907716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:02.907993 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:02.908203 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:02.908207 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:02.908956 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:02.908957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:02.909796 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:02.909798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:02.910216 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:02.910216 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:02.910355 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:02.910922 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:02.910930 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:02.912398 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:02.914142 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:02.914151 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:02.914154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:02.914471 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:02.914765 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:02.914766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:02.914767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:02.914923 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:02.915330 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:02.915336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:02.915470 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 19:08:02.915568 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 19:08:02.915715 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":99,"threads_started":1}
I20260620 19:08:02.917001 3427 heartbeater.cc:347] Connected to a master server at 127.0.203.62:39645
I20260620 19:08:02.917107 3427 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:02.917330 3427 heartbeater.cc:511] Master 127.0.203.62:39645 requested a full tablet report, sending...
I20260620 19:08:02.917943 3236 ts_manager.cc:194] Registered new tserver with Master: 4579f9d93d2f4ec7afdec3c47c3ff720 (127.0.203.1:44513)
I20260620 19:08:02.918931 3236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:38641
WARNING: no policy specified for kudu/127.0.203.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
I20260620 19:08:02.939910 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:39645
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:03.022526 3432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:03.022765 3432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:03.022791 3432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:03.024097 3432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:03.024144 3432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:03.024173 3432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:08:03.025727 3432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:03.025933 3432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:03.026130 3432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:03.031358 3438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:03.031340 3437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:03.031482 3432 server_base.cc:1034] running on GCE node
W20260620 19:08:03.031340 3440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:03.031775 3432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:03.032060 3432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:03.033231 3432 hybrid_clock.cc:648] HybridClock initialized: now 1781982483033211 us; error 33 us; skew 500 ppm
Jun 20 19:08:03 dist-test-slave-wd30 krb5kdc[3191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:03.035965 3432 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:08:03.036475 3432 webserver.cc:460] Webserver started at http://127.0.203.2:41569/ using document root <none> and password file <none>
I20260620 19:08:03.036717 3432 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:03.036765 3432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:03.036898 3432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:03.038004 3432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "710edd080193445c847634bfa6fb854d"
format_stamp: "Formatted at 2026-06-20 19:08:03 on dist-test-slave-wd30"
server_key: "c236c09d08631fcf7e4d604fa4a2b72d"
server_key_iv: "a60d4c2449a148075c7a06da5fc1931f"
server_key_version: "encryptionkey@0"
I20260620 19:08:03.038384 3432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "710edd080193445c847634bfa6fb854d"
format_stamp: "Formatted at 2026-06-20 19:08:03 on dist-test-slave-wd30"
server_key: "c236c09d08631fcf7e4d604fa4a2b72d"
server_key_iv: "a60d4c2449a148075c7a06da5fc1931f"
server_key_version: "encryptionkey@0"
I20260620 19:08:03.039866 3432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:08:03.040673 3447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:03.040900 3432 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:03.041010 3432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "710edd080193445c847634bfa6fb854d"
format_stamp: "Formatted at 2026-06-20 19:08:03 on dist-test-slave-wd30"
server_key: "c236c09d08631fcf7e4d604fa4a2b72d"
server_key_iv: "a60d4c2449a148075c7a06da5fc1931f"
server_key_version: "encryptionkey@0"
I20260620 19:08:03.041087 3432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:03.067972 3432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:03.069911 3432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:03.070077 3432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:03.070344 3432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:03.070639 3432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:03.070691 3432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:03.070726 3432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:03.070752 3432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:03.076714 3432 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:46017
I20260620 19:08:03.076848 3560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:46017 every 8 connection(s)
I20260620 19:08:03.077162 3432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:08:03 dist-test-slave-wd30 krb5kdc[3191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:03.085546 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 3432
I20260620 19:08:03.085625 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:08:03.085829 812 external_mini_cluster.cc:1428] Setting key e81ceab7224935e554674a658e889d07
I20260620 19:08:03.086824 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:03.077909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:34387 (local address 127.0.203.62:39645)
0620 19:08:03.077976 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:03.077979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:03.078174 (+ 195us) server_negotiation.cc:407] Connection header received
0620 19:08:03.079197 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:03.079198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:03.079210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:03.079274 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:03.079924 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:03.080406 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:03.081388 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:03.081593 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:03.081645 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:03.083772 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:03.083774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:03.083774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:03.083781 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:03.085306 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:03.085740 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:03.085742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:03.085743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:03.085801 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:03.086085 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:03.086086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:03.086087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:03.086254 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:03.086328 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:03.086653 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 19:08:03.086735 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:08:03.087019 3563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:03.077903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:39645 (local address 127.0.203.2:34387)
0620 19:08:03.078056 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:03.078090 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:03.079045 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:03.079365 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:03.079368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:03.079662 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:03.079850 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:03.079855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:03.080482 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:03.080482 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:03.081319 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:03.081321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:03.081666 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:03.081666 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:03.081761 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:03.082213 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:03.082221 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:03.083666 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:03.085411 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:03.085416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:03.085418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:03.085648 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:03.085867 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:03.085868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:03.085868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:03.085994 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:03.086328 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:03.086336 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:03.086495 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 19:08:03.086612 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 19:08:03.086772 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260620 19:08:03.087540 3561 heartbeater.cc:347] Connected to a master server at 127.0.203.62:39645
I20260620 19:08:03.087632 3561 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:03.087842 3561 heartbeater.cc:511] Master 127.0.203.62:39645 requested a full tablet report, sending...
I20260620 19:08:03.088289 3236 ts_manager.cc:194] Registered new tserver with Master: 710edd080193445c847634bfa6fb854d (127.0.203.2:46017)
I20260620 19:08:03.088685 3236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:34387
WARNING: no policy specified for kudu/127.0.203.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
I20260620 19:08:03.110219 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:39645
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:03.191859 3566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:03.192051 3566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:03.192073 3566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:03.193189 3566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:03.193230 3566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:03.193281 3566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:08:03.194747 3566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:03.194964 3566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:03.195190 3566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:03.200309 3572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:03.200359 3574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:03.200711 3566 server_base.cc:1034] running on GCE node
W20260620 19:08:03.200359 3571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:03.201067 3566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:03.201368 3566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:03.202551 3566 hybrid_clock.cc:648] HybridClock initialized: now 1781982483202537 us; error 51 us; skew 500 ppm
Jun 20 19:08:03 dist-test-slave-wd30 krb5kdc[3191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:03.205446 3566 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:08:03.205929 3566 webserver.cc:460] Webserver started at http://127.0.203.3:42603/ using document root <none> and password file <none>
I20260620 19:08:03.206173 3566 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:03.206244 3566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:03.206396 3566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:03.207594 3566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "d4a15a30f0304a21ac364241d5efcec3"
format_stamp: "Formatted at 2026-06-20 19:08:03 on dist-test-slave-wd30"
server_key: "f0ef29325fa4a0c5515305ddebe8d2f1"
server_key_iv: "e8d8a13746404668e0ed0c5ab5103955"
server_key_version: "encryptionkey@0"
I20260620 19:08:03.207983 3566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "d4a15a30f0304a21ac364241d5efcec3"
format_stamp: "Formatted at 2026-06-20 19:08:03 on dist-test-slave-wd30"
server_key: "f0ef29325fa4a0c5515305ddebe8d2f1"
server_key_iv: "e8d8a13746404668e0ed0c5ab5103955"
server_key_version: "encryptionkey@0"
I20260620 19:08:03.209460 3566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:08:03.210294 3581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:03.210487 3566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:03.210608 3566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "d4a15a30f0304a21ac364241d5efcec3"
format_stamp: "Formatted at 2026-06-20 19:08:03 on dist-test-slave-wd30"
server_key: "f0ef29325fa4a0c5515305ddebe8d2f1"
server_key_iv: "e8d8a13746404668e0ed0c5ab5103955"
server_key_version: "encryptionkey@0"
I20260620 19:08:03.210688 3566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:03.223263 3566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:03.226002 3566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:03.226204 3566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:03.226466 3566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:03.226820 3566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:03.226879 3566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:03.226923 3566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:03.226951 3566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:03.233489 3566 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:43657
I20260620 19:08:03.233552 3694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:43657 every 8 connection(s)
I20260620 19:08:03.233954 3566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:08:03.235700 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 3566
I20260620 19:08:03.235772 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:08:03.235960 812 external_mini_cluster.cc:1428] Setting key dac50318758e8aef7b792ff7c1c2f8db
Jun 20 19:08:03 dist-test-slave-wd30 krb5kdc[3191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:03.244575 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:03.234739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:38461 (local address 127.0.203.62:39645)
0620 19:08:03.234784 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:03.234786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:03.234991 (+ 205us) server_negotiation.cc:407] Connection header received
0620 19:08:03.236137 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:03.236138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:03.236154 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:03.236256 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:03.236939 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:03.237746 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:03.239056 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:03.239331 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:03.239405 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:03.241711 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:03.241713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:03.241713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:03.241721 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:03.242991 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:03.243466 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:03.243468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:03.243468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:03.243534 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:03.243808 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:03.243809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:03.243810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:03.243996 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:03.244075 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:03.244379 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 19:08:03.244467 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:08:03.244797 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:03.234697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:39645 (local address 127.0.203.3:38461)
0620 19:08:03.234858 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:03.234903 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:03.235955 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:03.236357 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:03.236361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:03.236625 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:03.236824 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:03.236829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:03.237857 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:03.237860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:03.238955 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:03.238958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:03.239433 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:03.239435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:03.239576 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:03.240163 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:03.240172 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:03.241600 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:03.243086 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:03.243094 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:03.243097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:03.243369 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:03.243617 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:03.243618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:03.243618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:03.243735 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:03.244081 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:03.244088 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:03.244237 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 19:08:03.244347 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 19:08:03.244520 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260620 19:08:03.245411 3695 heartbeater.cc:347] Connected to a master server at 127.0.203.62:39645
I20260620 19:08:03.245501 3695 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:03.245657 3695 heartbeater.cc:511] Master 127.0.203.62:39645 requested a full tablet report, sending...
I20260620 19:08:03.246001 3236 ts_manager.cc:194] Registered new tserver with Master: d4a15a30f0304a21ac364241d5efcec3 (127.0.203.3:43657)
I20260620 19:08:03.246414 3236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:38461
I20260620 19:08:03.247663 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 19:08:03.259358 3275 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:03.258234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36204 (local address 127.0.203.62:39645)
0620 19:08:03.258324 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:03.258327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:03.258431 (+ 104us) server_negotiation.cc:407] Connection header received
0620 19:08:03.258523 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:03.258524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:03.258538 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:03.258611 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:03.259270 (+ 659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36204: BlockingRecv error: recv got EOF from 127.0.0.1:36204 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:08:03.260164 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 3298
I20260620 19:08:03.264335 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 3432
I20260620 19:08:03.268292 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 3566
I20260620 19:08:03.273401 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 3204
2026-06-20T19:08:03Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2213 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:08:03 dist-test-slave-wd30 krb5kdc[3707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:08:03 dist-test-slave-wd30 krb5kdc[3707](info): set up 2 sockets
Jun 20 19:08:03 dist-test-slave-wd30 krb5kdc[3707](info): commencing operation
krb5kdc: starting...
W20260620 19:08:04.319008 812 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:08:04 dist-test-slave-wd30 krb5kdc[3707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:08:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:08:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:08:04.437773 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:36257
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:36257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:04.540340 3720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:04.540697 3720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:04.540748 3720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:04.543084 3720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:04.543176 3720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:04.543198 3720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:04.543222 3720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:04.543246 3720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:08:04.546209 3720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:36257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:36257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:04.546689 3720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:04.547000 3720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:04.552873 3725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:04.552871 3728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:04.552896 3726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:04.553370 3720 server_base.cc:1034] running on GCE node
I20260620 19:08:04.553725 3720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:04.554090 3720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:04.555251 3720 hybrid_clock.cc:648] HybridClock initialized: now 1781982484555220 us; error 47 us; skew 500 ppm
Jun 20 19:08:04 dist-test-slave-wd30 krb5kdc[3707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:04.561086 3720 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:08:04.561599 3720 webserver.cc:460] Webserver started at http://127.0.203.62:34833/ using document root <none> and password file <none>
I20260620 19:08:04.561858 3720 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:04.561942 3720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:04.562093 3720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:04.563313 3720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "53cd2b03473e46a98515e6a98dfae49a"
format_stamp: "Formatted at 2026-06-20 19:08:04 on dist-test-slave-wd30"
server_key: "41bd6404f563d3bb73b5bb4819283e62"
server_key_iv: "d62dc524a292ed83c843884cb268201f"
server_key_version: "encryptionkey@0"
I20260620 19:08:04.563694 3720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "53cd2b03473e46a98515e6a98dfae49a"
format_stamp: "Formatted at 2026-06-20 19:08:04 on dist-test-slave-wd30"
server_key: "41bd6404f563d3bb73b5bb4819283e62"
server_key_iv: "d62dc524a292ed83c843884cb268201f"
server_key_version: "encryptionkey@0"
I20260620 19:08:04.565078 3720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:08:04.565946 3735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:04.566167 3720 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:08:04.566283 3720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "53cd2b03473e46a98515e6a98dfae49a"
format_stamp: "Formatted at 2026-06-20 19:08:04 on dist-test-slave-wd30"
server_key: "41bd6404f563d3bb73b5bb4819283e62"
server_key_iv: "d62dc524a292ed83c843884cb268201f"
server_key_version: "encryptionkey@0"
I20260620 19:08:04.566360 3720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:04.575229 3720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:04.577267 3720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:04.577436 3720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:04.581674 3720 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:36257
I20260620 19:08:04.581725 3787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:36257 every 8 connection(s)
I20260620 19:08:04.582113 3720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:04.582974 3788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:04.583281 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 3720
I20260620 19:08:04.583385 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:08:04.583557 812 external_mini_cluster.cc:1428] Setting key 6b974e2edf49f991599f916233021448
I20260620 19:08:04.586023 3788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a: Bootstrap starting.
I20260620 19:08:04.586663 3788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:04.586998 3788 log.cc:824] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:04.587859 3788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a: No bootstrap required, opened a new log
I20260620 19:08:04.590420 3788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cd2b03473e46a98515e6a98dfae49a" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36257 } }
I20260620 19:08:04.590569 3788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:04.590591 3788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53cd2b03473e46a98515e6a98dfae49a, State: Initialized, Role: FOLLOWER
Jun 20 19:08:04 dist-test-slave-wd30 krb5kdc[3707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:04.590729 3788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cd2b03473e46a98515e6a98dfae49a" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36257 } }
I20260620 19:08:04.590802 3788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:04.590833 3788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:04.590885 3788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:04.591542 3788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cd2b03473e46a98515e6a98dfae49a" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36257 } }
I20260620 19:08:04.591670 3788 leader_election.cc:302] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53cd2b03473e46a98515e6a98dfae49a; no voters:
I20260620 19:08:04.591876 3788 leader_election.cc:288] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:08:04.592152 3788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:04.592371 3793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:04.592522 3793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a [term 1 LEADER]: Becoming Leader. State: Replica: 53cd2b03473e46a98515e6a98dfae49a, State: Running, Role: LEADER
I20260620 19:08:04.592706 3793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cd2b03473e46a98515e6a98dfae49a" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36257 } }
I20260620 19:08:04.593452 3794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53cd2b03473e46a98515e6a98dfae49a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cd2b03473e46a98515e6a98dfae49a" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36257 } } }
I20260620 19:08:04.594370 3794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:04.594441 3795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53cd2b03473e46a98515e6a98dfae49a. Latest consensus state: current_term: 1 leader_uuid: "53cd2b03473e46a98515e6a98dfae49a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cd2b03473e46a98515e6a98dfae49a" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36257 } } }
W20260620 19:08:04.594002 3807 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 19:08:04.594527 3795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a [sys.catalog]: This master's current role is: LEADER
W20260620 19:08:04.594568 3807 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:08:04.594805 3810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:08:04.595007 3810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:08:04.595595 3791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:04.584108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43000 (local address 127.0.203.62:36257)
0620 19:08:04.584284 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:04.584294 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:04.584327 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:08:04.584740 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:04.584746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:04.584793 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:04.584911 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:04.585730 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:04.586649 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:04.587673 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:04.588215 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:04.588318 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:04.591331 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:04.591335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:04.591335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:04.591347 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:04.593770 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:04.594258 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:04.594261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:04.594261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:04.594384 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:04.594624 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:04.594625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:04.594625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:04.594986 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:04.595188 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:04.595275 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 19:08:04.595418 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260620 19:08:04.596638 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:08:04.596778 3810 catalog_manager.cc:1347] Generated new cluster ID: c50bb469e3b3457499964c10f04834c3
I20260620 19:08:04.596827 3810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:08:04.606313 3810 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:08:04.606874 3810 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:04.612957 3810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53cd2b03473e46a98515e6a98dfae49a: Generated new TSK 0
I20260620 19:08:04.613148 3810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:08:04.620428 3791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:04.616218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.0.203.62:36257)
0620 19:08:04.616302 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:04.616304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:04.616474 (+ 170us) server_negotiation.cc:407] Connection header received
0620 19:08:04.616511 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:04.616513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:04.616524 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:04.616610 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:04.616995 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:04.617794 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:04.618657 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:04.618988 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:04.619059 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:04.619867 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:04.619869 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:04.619869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:04.620108 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:04.620129 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:04.620136 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:04.620201 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:04.620313 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 19:08:04.620342 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:08:04.622797 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 3720
2026-06-20T19:08:04Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1347 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:08:04 dist-test-slave-wd30 krb5kdc[3821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:08:04 dist-test-slave-wd30 krb5kdc[3821](info): set up 2 sockets
Jun 20 19:08:04 dist-test-slave-wd30 krb5kdc[3821](info): commencing operation
krb5kdc: starting...
W20260620 19:08:05.659019 812 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:08:05 dist-test-slave-wd30 krb5kdc[3821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:08:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:08:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:08:05.776087 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:33885
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:33885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:05.857434 3834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:05.857712 3834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:05.857751 3834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:05.858986 3834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:05.859066 3834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:05.859089 3834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:05.859112 3834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:05.859194 3834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:08:05.860805 3834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:33885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:33885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:05.861116 3834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:05.861357 3834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:05.866461 3842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:05.866461 3840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:05.866464 3839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:05.866647 3834 server_base.cc:1034] running on GCE node
I20260620 19:08:05.867035 3834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:05.867403 3834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:05.868566 3834 hybrid_clock.cc:648] HybridClock initialized: now 1781982485868543 us; error 56 us; skew 500 ppm
Jun 20 19:08:05 dist-test-slave-wd30 krb5kdc[3821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:05.871287 3834 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:08:05.871712 3834 webserver.cc:460] Webserver started at http://127.0.203.62:42659/ using document root <none> and password file <none>
I20260620 19:08:05.871898 3834 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:05.871939 3834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:05.872038 3834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:05.873073 3834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "3482d90c559b4e46a570fbb84790a1d1"
format_stamp: "Formatted at 2026-06-20 19:08:05 on dist-test-slave-wd30"
server_key: "b2461082aa7e62179d385677e111b7a5"
server_key_iv: "2fd2a6ac85cfeda1a85f22fef2b715ec"
server_key_version: "encryptionkey@0"
I20260620 19:08:05.873373 3834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "3482d90c559b4e46a570fbb84790a1d1"
format_stamp: "Formatted at 2026-06-20 19:08:05 on dist-test-slave-wd30"
server_key: "b2461082aa7e62179d385677e111b7a5"
server_key_iv: "2fd2a6ac85cfeda1a85f22fef2b715ec"
server_key_version: "encryptionkey@0"
I20260620 19:08:05.874645 3834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:08:05.875373 3849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:05.875553 3834 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:05.875655 3834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "3482d90c559b4e46a570fbb84790a1d1"
format_stamp: "Formatted at 2026-06-20 19:08:05 on dist-test-slave-wd30"
server_key: "b2461082aa7e62179d385677e111b7a5"
server_key_iv: "2fd2a6ac85cfeda1a85f22fef2b715ec"
server_key_version: "encryptionkey@0"
I20260620 19:08:05.875741 3834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:05.882124 3834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:05.884563 3834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:05.884725 3834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:05.888695 3834 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:33885
I20260620 19:08:05.888728 3901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:33885 every 8 connection(s)
I20260620 19:08:05.889109 3834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:05.890453 3902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:05.890718 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 3834
I20260620 19:08:05.890820 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:08:05.891001 812 external_mini_cluster.cc:1428] Setting key 986c3aa88054483db7127c5dcb3b9d8f
I20260620 19:08:05.893852 3902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1: Bootstrap starting.
I20260620 19:08:05.894637 3902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:05.894937 3902 log.cc:824] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:05.895669 3902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1: No bootstrap required, opened a new log
Jun 20 19:08:05 dist-test-slave-wd30 krb5kdc[3821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:05.897712 3902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3482d90c559b4e46a570fbb84790a1d1" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 33885 } }
I20260620 19:08:05.897891 3902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:05.897939 3902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3482d90c559b4e46a570fbb84790a1d1, State: Initialized, Role: FOLLOWER
I20260620 19:08:05.898095 3902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3482d90c559b4e46a570fbb84790a1d1" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 33885 } }
I20260620 19:08:05.898204 3902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:05.898249 3902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:05.898299 3902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:05.899060 3902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3482d90c559b4e46a570fbb84790a1d1" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 33885 } }
I20260620 19:08:05.899192 3902 leader_election.cc:302] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3482d90c559b4e46a570fbb84790a1d1; no voters:
I20260620 19:08:05.899386 3902 leader_election.cc:288] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:08:05.899434 3907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:05.899593 3907 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1 [term 1 LEADER]: Becoming Leader. State: Replica: 3482d90c559b4e46a570fbb84790a1d1, State: Running, Role: LEADER
I20260620 19:08:05.899686 3902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:05.899724 3907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3482d90c559b4e46a570fbb84790a1d1" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 33885 } }
I20260620 19:08:05.900030 3907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3482d90c559b4e46a570fbb84790a1d1. Latest consensus state: current_term: 1 leader_uuid: "3482d90c559b4e46a570fbb84790a1d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3482d90c559b4e46a570fbb84790a1d1" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 33885 } } }
I20260620 19:08:05.900125 3907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1 [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:05.900049 3908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3482d90c559b4e46a570fbb84790a1d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3482d90c559b4e46a570fbb84790a1d1" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 33885 } } }
I20260620 19:08:05.900261 3908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1 [sys.catalog]: This master's current role is: LEADER
W20260620 19:08:05.901538 3923 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:08:05.901595 3923 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:08:05.901647 3914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:08:05.901801 3914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:08:05.902033 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:05.891544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45396 (local address 127.0.203.62:33885)
0620 19:08:05.891712 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:05.891720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:05.891746 (+ 26us) server_negotiation.cc:407] Connection header received
0620 19:08:05.892178 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:05.892185 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:05.892235 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:05.892349 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:05.893136 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:05.894065 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:05.895186 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:05.895687 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:05.895815 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:05.898189 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:05.898192 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:05.898192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:05.898204 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:05.900281 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:05.900652 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:05.900656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:05.900656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:05.900764 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:05.900989 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:05.900991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:05.900991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:05.901340 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:05.901517 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:05.901637 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 19:08:05.901780 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260620 19:08:05.903513 3914 catalog_manager.cc:1347] Generated new cluster ID: a6f62ed1000d4f38b8f3db76caa0dac8
I20260620 19:08:05.903566 3914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:08:05.914335 3914 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:08:05.914863 3914 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:05.919739 3914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3482d90c559b4e46a570fbb84790a1d1: Generated new TSK 0
I20260620 19:08:05.919943 3914 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
I20260620 19:08:05.952450 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:33885
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:06.057793 3928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:06.058029 3928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:06.058058 3928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:06.059504 3928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:06.059572 3928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:06.059603 3928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:08:06.061357 3928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:33885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:06.061717 3928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:06.062016 3928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:06.067472 3933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:06.067456 3936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:06.067461 3934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:06.067807 3928 server_base.cc:1034] running on GCE node
I20260620 19:08:06.068073 3928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:06.068363 3928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:06.069613 3928 hybrid_clock.cc:648] HybridClock initialized: now 1781982486069597 us; error 37 us; skew 500 ppm
Jun 20 19:08:06 dist-test-slave-wd30 krb5kdc[3821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:06.072934 3928 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:08:06.073583 3928 webserver.cc:460] Webserver started at http://127.0.203.1:33647/ using document root <none> and password file <none>
I20260620 19:08:06.073892 3928 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:06.073983 3928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:06.074144 3928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:06.075446 3928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "a024b1c661d7454d977e52e3b82d67ba"
format_stamp: "Formatted at 2026-06-20 19:08:06 on dist-test-slave-wd30"
server_key: "f0987e13e6143f959f9b0c65f823adae"
server_key_iv: "97b9d232c7167b2683c8382773e8becf"
server_key_version: "encryptionkey@0"
I20260620 19:08:06.075861 3928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "a024b1c661d7454d977e52e3b82d67ba"
format_stamp: "Formatted at 2026-06-20 19:08:06 on dist-test-slave-wd30"
server_key: "f0987e13e6143f959f9b0c65f823adae"
server_key_iv: "97b9d232c7167b2683c8382773e8becf"
server_key_version: "encryptionkey@0"
I20260620 19:08:06.077401 3928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:08:06.078284 3943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:06.078510 3928 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:06.078619 3928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "a024b1c661d7454d977e52e3b82d67ba"
format_stamp: "Formatted at 2026-06-20 19:08:06 on dist-test-slave-wd30"
server_key: "f0987e13e6143f959f9b0c65f823adae"
server_key_iv: "97b9d232c7167b2683c8382773e8becf"
server_key_version: "encryptionkey@0"
I20260620 19:08:06.078703 3928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:06.092746 3928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:06.094705 3928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:06.094882 3928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:06.095149 3928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:06.095521 3928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:06.095583 3928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:06.095625 3928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:06.095654 3928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:06.101964 3928 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:46151
I20260620 19:08:06.102015 4056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:46151 every 8 connection(s)
I20260620 19:08:06.102353 3928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:08:06 dist-test-slave-wd30 krb5kdc[3821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:06.109325 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 3928
I20260620 19:08:06.109440 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:08:06.109642 812 external_mini_cluster.cc:1428] Setting key dab25439cc3e15bfb5b1264fd2098784
I20260620 19:08:06.112587 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:06.103088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:48943 (local address 127.0.203.62:33885)
0620 19:08:06.103182 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:06.103184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:06.103456 (+ 272us) server_negotiation.cc:407] Connection header received
0620 19:08:06.103935 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:06.103936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:06.103947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:06.104005 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:06.104947 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:06.105439 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:06.106413 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:06.106627 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:06.106682 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:06.109020 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:06.109022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:06.109022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:06.109029 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:06.110754 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:06.111383 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:06.111385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:06.111385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:06.111445 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:06.111763 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:06.111764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:06.111764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:06.111972 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:06.112046 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:06.112358 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 19:08:06.112478 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 19:08:06.112614 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:06.103075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:33885 (local address 127.0.203.1:48943)
0620 19:08:06.103318 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:06.103358 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:06.103852 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:06.104062 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:06.104065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:06.104661 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:06.104862 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:06.104867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:06.105528 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:06.105529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:06.106333 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:06.106335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:06.106703 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:06.106704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:06.106810 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:06.107455 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:06.107464 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:06.108929 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:06.110847 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:06.110857 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:06.110859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:06.111292 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:06.111509 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:06.111510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:06.111510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:06.111674 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:06.112039 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:06.112044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:06.112176 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 19:08:06.112299 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 19:08:06.112438 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":135,"threads_started":1}
I20260620 19:08:06.113204 4057 heartbeater.cc:347] Connected to a master server at 127.0.203.62:33885
I20260620 19:08:06.113317 4057 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:06.113551 4057 heartbeater.cc:511] Master 127.0.203.62:33885 requested a full tablet report, sending...
I20260620 19:08:06.114138 3866 ts_manager.cc:194] Registered new tserver with Master: a024b1c661d7454d977e52e3b82d67ba (127.0.203.1:46151)
I20260620 19:08:06.115162 3866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:48943
WARNING: no policy specified for kudu/127.0.203.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
I20260620 19:08:06.135937 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:33885
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:06.227078 4062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:06.227468 4062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:06.227497 4062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:06.228847 4062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:06.228907 4062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:06.228941 4062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:08:06.230626 4062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:33885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:06.230865 4062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:06.231073 4062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:06.236665 4068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:06.236685 4070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:06.236749 4067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:06.236907 4062 server_base.cc:1034] running on GCE node
I20260620 19:08:06.237206 4062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:06.237478 4062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:06.238668 4062 hybrid_clock.cc:648] HybridClock initialized: now 1781982486238651 us; error 31 us; skew 500 ppm
Jun 20 19:08:06 dist-test-slave-wd30 krb5kdc[3821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:06.241458 4062 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:08:06.241925 4062 webserver.cc:460] Webserver started at http://127.0.203.2:40531/ using document root <none> and password file <none>
I20260620 19:08:06.242177 4062 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:06.242247 4062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:06.242396 4062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:06.243566 4062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "c9b90c79adae45cdbbeeebd5fcae0004"
format_stamp: "Formatted at 2026-06-20 19:08:06 on dist-test-slave-wd30"
server_key: "74af58bc8830c604f952220c9ed4ca1e"
server_key_iv: "b2f630cade2b1e358bed256a7fbdf9be"
server_key_version: "encryptionkey@0"
I20260620 19:08:06.243937 4062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "c9b90c79adae45cdbbeeebd5fcae0004"
format_stamp: "Formatted at 2026-06-20 19:08:06 on dist-test-slave-wd30"
server_key: "74af58bc8830c604f952220c9ed4ca1e"
server_key_iv: "b2f630cade2b1e358bed256a7fbdf9be"
server_key_version: "encryptionkey@0"
I20260620 19:08:06.245329 4062 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 19:08:06.246135 4077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:06.246366 4062 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 19:08:06.246476 4062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "c9b90c79adae45cdbbeeebd5fcae0004"
format_stamp: "Formatted at 2026-06-20 19:08:06 on dist-test-slave-wd30"
server_key: "74af58bc8830c604f952220c9ed4ca1e"
server_key_iv: "b2f630cade2b1e358bed256a7fbdf9be"
server_key_version: "encryptionkey@0"
I20260620 19:08:06.246564 4062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:06.268934 4062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:06.271440 4062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:06.271615 4062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:06.271855 4062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:06.272184 4062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:06.272243 4062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:06.272287 4062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:06.272315 4062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:06.278472 4062 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:45005
I20260620 19:08:06.278555 4190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:45005 every 8 connection(s)
I20260620 19:08:06.278945 4062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:08:06.281286 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 4062
I20260620 19:08:06.281404 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:08:06.281608 812 external_mini_cluster.cc:1428] Setting key 5e857296a21aec2ed3780826b4fee034
Jun 20 19:08:06 dist-test-slave-wd30 krb5kdc[3821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:06.290599 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:06.279730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:54323 (local address 127.0.203.62:33885)
0620 19:08:06.279777 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:06.279779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:06.280039 (+ 260us) server_negotiation.cc:407] Connection header received
0620 19:08:06.280598 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:06.280599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:06.280612 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:06.280666 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:06.281989 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:06.282726 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:06.284619 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:06.284944 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:06.285018 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:06.287537 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:06.287539 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:06.287539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:06.287549 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:06.289203 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:06.289604 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:06.289606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:06.289607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:06.289672 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:06.289907 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:06.289908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:06.289909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:06.290069 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:06.290144 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:06.290397 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 19:08:06.290488 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:08:06.290693 4193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:06.279738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:33885 (local address 127.0.203.2:54323)
0620 19:08:06.279911 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:06.279948 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:06.280475 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:06.280746 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:06.280750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:06.281531 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:06.281853 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:06.281859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:06.282887 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:06.282889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:06.284463 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:06.284466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:06.285054 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:06.285055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:06.285212 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:06.285895 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:06.285905 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:06.287428 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:06.289291 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:06.289298 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:06.289300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:06.289525 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:06.289744 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:06.289744 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:06.289744 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:06.289843 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:06.290136 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:06.290140 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:06.290263 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 19:08:06.290347 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:08:06.290471 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260620 19:08:06.291203 4191 heartbeater.cc:347] Connected to a master server at 127.0.203.62:33885
I20260620 19:08:06.291313 4191 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.203.3@KRBTEST.COM; defaulting to no policy
I20260620 19:08:06.291493 4191 heartbeater.cc:511] Master 127.0.203.62:33885 requested a full tablet report, sending...
I20260620 19:08:06.291917 3866 ts_manager.cc:194] Registered new tserver with Master: c9b90c79adae45cdbbeeebd5fcae0004 (127.0.203.2:45005)
I20260620 19:08:06.292400 3866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:54323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
I20260620 19:08:06.309515 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:33885
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:06.392076 4196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:06.392292 4196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:06.392315 4196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:06.393445 4196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:06.393481 4196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:06.393507 4196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:08:06.394891 4196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:33885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:06.395074 4196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:06.395340 4196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:08:06.400585 4196 server_base.cc:1034] running on GCE node
W20260620 19:08:06.400569 4204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:06.400569 4202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:06.400619 4201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:06.401026 4196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:06.401350 4196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:06.402508 4196 hybrid_clock.cc:648] HybridClock initialized: now 1781982486402505 us; error 35 us; skew 500 ppm
Jun 20 19:08:06 dist-test-slave-wd30 krb5kdc[3821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:06.405268 4196 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:08:06.405670 4196 webserver.cc:460] Webserver started at http://127.0.203.3:34817/ using document root <none> and password file <none>
I20260620 19:08:06.405853 4196 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:06.405891 4196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:06.405980 4196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:06.407274 4196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "6d17338d89b24e21ac8c3cf53b52ee86"
format_stamp: "Formatted at 2026-06-20 19:08:06 on dist-test-slave-wd30"
server_key: "87a6e178904817e8e963c2d75653beef"
server_key_iv: "3e57843de102bc022b108931378ac882"
server_key_version: "encryptionkey@0"
I20260620 19:08:06.407758 4196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "6d17338d89b24e21ac8c3cf53b52ee86"
format_stamp: "Formatted at 2026-06-20 19:08:06 on dist-test-slave-wd30"
server_key: "87a6e178904817e8e963c2d75653beef"
server_key_iv: "3e57843de102bc022b108931378ac882"
server_key_version: "encryptionkey@0"
I20260620 19:08:06.409209 4196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:08:06.410053 4211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:06.410293 4196 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:06.410413 4196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "6d17338d89b24e21ac8c3cf53b52ee86"
format_stamp: "Formatted at 2026-06-20 19:08:06 on dist-test-slave-wd30"
server_key: "87a6e178904817e8e963c2d75653beef"
server_key_iv: "3e57843de102bc022b108931378ac882"
server_key_version: "encryptionkey@0"
I20260620 19:08:06.410514 4196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:06.423349 4196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:06.425985 4196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:06.426169 4196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:06.426441 4196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:06.426777 4196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:06.426836 4196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:06.426913 4196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:06.426949 4196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:06.433915 4196 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:38253
I20260620 19:08:06.433971 4324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:38253 every 8 connection(s)
I20260620 19:08:06.434406 4196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:08:06.434688 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 4196
I20260620 19:08:06.434791 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:08:06.435026 812 external_mini_cluster.cc:1428] Setting key ad8ccb52ba623dc2c349e8fd7c7994c5
Jun 20 19:08:06 dist-test-slave-wd30 krb5kdc[3821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:06.444919 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:06.435302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:50291 (local address 127.0.203.62:33885)
0620 19:08:06.435355 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:06.435358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:06.435760 (+ 402us) server_negotiation.cc:407] Connection header received
0620 19:08:06.436464 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:06.436465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:06.436477 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:06.436520 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:06.437583 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:06.438104 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:06.439301 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:06.439505 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:06.439565 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:06.441734 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:06.441735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:06.441735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:06.441742 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:06.443334 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:06.443788 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:06.443789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:06.443790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:06.443851 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:06.444271 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:06.444271 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:06.444272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:06.444400 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:06.444468 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:06.444775 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 19:08:06.444846 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:08:06.445196 4327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:06.435335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:33885 (local address 127.0.203.3:50291)
0620 19:08:06.435560 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:06.435618 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:06.436369 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:06.436591 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:06.436594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:06.437249 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:06.437464 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:06.437468 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:06.438211 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:06.438212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:06.439204 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:06.439208 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:06.439584 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:06.439585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:06.439694 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:06.440191 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:06.440200 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:06.441621 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:06.443417 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:06.443423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:06.443426 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:06.443695 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:06.444075 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:06.444076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:06.444076 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:06.444194 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:06.444507 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:06.444513 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:06.444647 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 19:08:06.444747 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 19:08:06.444891 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":89,"threads_started":1}
I20260620 19:08:06.445755 4325 heartbeater.cc:347] Connected to a master server at 127.0.203.62:33885
I20260620 19:08:06.445865 4325 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:06.446055 4325 heartbeater.cc:511] Master 127.0.203.62:33885 requested a full tablet report, sending...
I20260620 19:08:06.446535 3866 ts_manager.cc:194] Registered new tserver with Master: 6d17338d89b24e21ac8c3cf53b52ee86 (127.0.203.3:38253)
I20260620 19:08:06.447023 3866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:50291
I20260620 19:08:06.447368 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:08:06 dist-test-slave-wd30 krb5kdc[3821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982486, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 19:08:06 dist-test-slave-wd30 krb5kdc[3821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982486, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:06.471709 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:06.463638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60222 (local address 127.0.203.62:33885)
0620 19:08:06.463834 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:06.463835 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:06.463846 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:08:06.463910 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:06.463911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:06.463921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:06.463964 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:06.464649 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:06.465149 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:06.465974 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:06.466164 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:06.466231 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:06.468751 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:06.468753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:06.468754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:06.468762 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:06.470234 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:06.470827 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:06.470828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:06.470828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:06.470887 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:06.471105 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:06.471105 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:06.471105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:06.471328 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:06.471408 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:06.471536 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 19:08:06.471604 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
W20260620 19:08:06.471984 3866 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60222
I20260620 19:08:06.472442 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 3928
I20260620 19:08:06.477093 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 4062
I20260620 19:08:06.481451 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 4196
I20260620 19:08:06.485652 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 3834
2026-06-20T19:08:06Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1871 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:08:06 dist-test-slave-wd30 krb5kdc[4337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:08:06 dist-test-slave-wd30 krb5kdc[4337](info): set up 2 sockets
Jun 20 19:08:06 dist-test-slave-wd30 krb5kdc[4337](info): commencing operation
krb5kdc: starting...
W20260620 19:08:07.535018 812 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:08:07 dist-test-slave-wd30 krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:08:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:08:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:08:07.653856 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:34477
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:34477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:07.734951 4350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:07.735188 4350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:07.735210 4350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:07.736459 4350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:07.736503 4350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:07.736517 4350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:07.736529 4350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:07.736542 4350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:08:07.738118 4350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:34477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:34477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:07.738372 4350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:07.738602 4350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:07.743567 4358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:07.743578 4355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:07.743561 4356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:07.743637 4350 server_base.cc:1034] running on GCE node
I20260620 19:08:07.744025 4350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:07.744344 4350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:07.745522 4350 hybrid_clock.cc:648] HybridClock initialized: now 1781982487745517 us; error 37 us; skew 500 ppm
Jun 20 19:08:07 dist-test-slave-wd30 krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:07.748144 4350 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:08:07.748608 4350 webserver.cc:460] Webserver started at http://127.0.203.62:42765/ using document root <none> and password file <none>
I20260620 19:08:07.748798 4350 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:07.748840 4350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:07.748929 4350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:07.749965 4350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "57bd2e742878462ebe181abad99549aa"
format_stamp: "Formatted at 2026-06-20 19:08:07 on dist-test-slave-wd30"
server_key: "b1f88b8b82db3bfbee8e39ab6359d356"
server_key_iv: "c424f2f07545a71b9c65709ec6e53483"
server_key_version: "encryptionkey@0"
I20260620 19:08:07.750279 4350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "57bd2e742878462ebe181abad99549aa"
format_stamp: "Formatted at 2026-06-20 19:08:07 on dist-test-slave-wd30"
server_key: "b1f88b8b82db3bfbee8e39ab6359d356"
server_key_iv: "c424f2f07545a71b9c65709ec6e53483"
server_key_version: "encryptionkey@0"
I20260620 19:08:07.751617 4350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:08:07.752417 4365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:07.752681 4350 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:08:07.752765 4350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "57bd2e742878462ebe181abad99549aa"
format_stamp: "Formatted at 2026-06-20 19:08:07 on dist-test-slave-wd30"
server_key: "b1f88b8b82db3bfbee8e39ab6359d356"
server_key_iv: "c424f2f07545a71b9c65709ec6e53483"
server_key_version: "encryptionkey@0"
I20260620 19:08:07.752821 4350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:07.794234 4350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:07.796293 4350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:07.796408 4350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:07.800190 4350 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:34477
I20260620 19:08:07.800254 4417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:34477 every 8 connection(s)
I20260620 19:08:07.800578 4350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:07.801433 4418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:07.803818 4418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa: Bootstrap starting.
I20260620 19:08:07.804396 4418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:07.804702 4418 log.cc:824] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:07.805387 4418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa: No bootstrap required, opened a new log
I20260620 19:08:07.806946 4418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57bd2e742878462ebe181abad99549aa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 34477 } }
I20260620 19:08:07.807142 4418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:07.807215 4418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57bd2e742878462ebe181abad99549aa, State: Initialized, Role: FOLLOWER
I20260620 19:08:07.807387 4418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57bd2e742878462ebe181abad99549aa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 34477 } }
I20260620 19:08:07.807471 4418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:07.807515 4418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:07.807564 4418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:07.808115 4418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57bd2e742878462ebe181abad99549aa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 34477 } }
I20260620 19:08:07.808233 4418 leader_election.cc:302] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57bd2e742878462ebe181abad99549aa; no voters:
I20260620 19:08:07.808415 4418 leader_election.cc:288] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:08:07.808465 4421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:07.808672 4418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:07.808878 4421 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa [term 1 LEADER]: Becoming Leader. State: Replica: 57bd2e742878462ebe181abad99549aa, State: Running, Role: LEADER
I20260620 19:08:07.808974 4421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57bd2e742878462ebe181abad99549aa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 34477 } }
I20260620 19:08:07.809335 4423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57bd2e742878462ebe181abad99549aa. Latest consensus state: current_term: 1 leader_uuid: "57bd2e742878462ebe181abad99549aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57bd2e742878462ebe181abad99549aa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 34477 } } }
I20260620 19:08:07.809417 4423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:07.809489 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 4350
I20260620 19:08:07.809592 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:08:07.809753 4422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57bd2e742878462ebe181abad99549aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57bd2e742878462ebe181abad99549aa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 34477 } } }
I20260620 19:08:07.809808 4422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:07.809816 812 external_mini_cluster.cc:1428] Setting key 9bd2a1a1a8f111d1c4a413814973f97c
I20260620 19:08:07.810045 4430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:08:07.810660 4430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:08:07.812122 4430 catalog_manager.cc:1347] Generated new cluster ID: 9d972642c68e4542a54b4dc9dfcfb71c
I20260620 19:08:07.812175 4430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:08:07 dist-test-slave-wd30 krb5kdc[4337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:07.821348 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:07.810275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60440 (local address 127.0.203.62:34477)
0620 19:08:07.810472 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:07.810484 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:07.810514 (+ 30us) server_negotiation.cc:407] Connection header received
0620 19:08:07.810635 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:07.810643 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:07.810701 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:07.810854 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:07.811636 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:07.813084 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:07.814145 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:07.814755 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:07.814849 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:07.817639 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:07.817645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:07.817645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:07.817660 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:07.819693 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:07.820078 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:07.820082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:07.820082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:07.820192 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:07.820393 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:07.820393 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:07.820394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:07.820733 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:07.820889 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:07.820986 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 19:08:07.821127 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
I20260620 19:08:07.822651 4430 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:08:07.823227 4430 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:07.827301 4430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57bd2e742878462ebe181abad99549aa: Generated new TSK 0
I20260620 19:08:07.827486 4430 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
I20260620 19:08:07.858407 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:34477
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:07.941001 4444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:07.941190 4444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:07.941213 4444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:07.942299 4444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:07.942332 4444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:07.942358 4444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:08:07.943852 4444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:34477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:07.944038 4444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:07.944205 4444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:07.949126 4452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:07.949126 4449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:07.949122 4450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:07.949508 4444 server_base.cc:1034] running on GCE node
I20260620 19:08:07.949764 4444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:07.950003 4444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:07.951220 4444 hybrid_clock.cc:648] HybridClock initialized: now 1781982487951193 us; error 33 us; skew 500 ppm
Jun 20 19:08:07 dist-test-slave-wd30 krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:07.953867 4444 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:08:07.954321 4444 webserver.cc:460] Webserver started at http://127.0.203.1:35317/ using document root <none> and password file <none>
I20260620 19:08:07.954514 4444 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:07.954553 4444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:07.954640 4444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:07.955749 4444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "5718ce57c3d04e2f9f5c6f8f7f0cc458"
format_stamp: "Formatted at 2026-06-20 19:08:07 on dist-test-slave-wd30"
server_key: "a868f98aa46195ebc2173adc1d4f001b"
server_key_iv: "5bdfac8987beacc270efedaf8a1d664b"
server_key_version: "encryptionkey@0"
I20260620 19:08:07.956079 4444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "5718ce57c3d04e2f9f5c6f8f7f0cc458"
format_stamp: "Formatted at 2026-06-20 19:08:07 on dist-test-slave-wd30"
server_key: "a868f98aa46195ebc2173adc1d4f001b"
server_key_iv: "5bdfac8987beacc270efedaf8a1d664b"
server_key_version: "encryptionkey@0"
I20260620 19:08:07.957383 4444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:08:07.958093 4459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:07.958319 4444 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:07.958411 4444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "5718ce57c3d04e2f9f5c6f8f7f0cc458"
format_stamp: "Formatted at 2026-06-20 19:08:07 on dist-test-slave-wd30"
server_key: "a868f98aa46195ebc2173adc1d4f001b"
server_key_iv: "5bdfac8987beacc270efedaf8a1d664b"
server_key_version: "encryptionkey@0"
I20260620 19:08:07.958492 4444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:07.968108 4444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:07.970489 4444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:07.970652 4444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:07.970889 4444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:07.971295 4444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:07.971338 4444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:07.971362 4444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:07.971406 4444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:07.977290 4444 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:40623
I20260620 19:08:07.977336 4572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:40623 every 8 connection(s)
I20260620 19:08:07.977653 4444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:08:07.983319 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 4444
I20260620 19:08:07.983399 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:08:07.983600 812 external_mini_cluster.cc:1428] Setting key 8242d3a08e4bbfc1e83d10f637652a31
Jun 20 19:08:07 dist-test-slave-wd30 krb5kdc[4337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:07.987594 4575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:07.978360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:34477 (local address 127.0.203.1:44391)
0620 19:08:07.978516 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:07.978560 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:07.979569 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:07.979788 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:07.979791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:07.980056 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:07.980255 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:07.980258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:07.980911 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:07.980912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:07.981718 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:07.981720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:07.982071 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:07.982071 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:07.982198 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:07.982656 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:07.982664 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:07.984312 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:07.986204 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:07.986210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:07.986212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:07.986425 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:07.986635 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:07.986636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:07.986636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:07.986734 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:07.987015 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:07.987019 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:07.987210 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 19:08:07.987303 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 19:08:07.987428 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260620 19:08:07.987598 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:07.978365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:44391 (local address 127.0.203.62:34477)
0620 19:08:07.978444 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:07.978446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:07.978646 (+ 200us) server_negotiation.cc:407] Connection header received
0620 19:08:07.979659 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:07.979660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:07.979672 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:07.979731 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:07.980327 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:07.980832 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:07.981797 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:07.981996 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:07.982051 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:07.984406 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:07.984407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:07.984408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:07.984415 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:07.986103 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:07.986502 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:07.986504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:07.986504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:07.986566 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:07.986797 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:07.986798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:07.986798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:07.986947 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:07.987021 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:07.987357 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 19:08:07.987456 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:08:07.988656 4573 heartbeater.cc:347] Connected to a master server at 127.0.203.62:34477
I20260620 19:08:07.988780 4573 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:07.988960 4573 heartbeater.cc:511] Master 127.0.203.62:34477 requested a full tablet report, sending...
I20260620 19:08:07.989523 4382 ts_manager.cc:194] Registered new tserver with Master: 5718ce57c3d04e2f9f5c6f8f7f0cc458 (127.0.203.1:40623)
I20260620 19:08:07.990130 4382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:44391
WARNING: no policy specified for kudu/127.0.203.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
I20260620 19:08:08.009486 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:34477
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:08.091681 4578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:08.091892 4578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:08.091917 4578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:08.093087 4578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:08.093134 4578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:08.093163 4578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:08:08.094643 4578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:34477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:08.094892 4578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:08.095172 4578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:08.100160 4583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:08.100181 4586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:08.100168 4584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:08.100350 4578 server_base.cc:1034] running on GCE node
I20260620 19:08:08.100648 4578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:08.100847 4578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:08.102089 4578 hybrid_clock.cc:648] HybridClock initialized: now 1781982488102077 us; error 27 us; skew 500 ppm
Jun 20 19:08:08 dist-test-slave-wd30 krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:08.104677 4578 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:08:08.105149 4578 webserver.cc:460] Webserver started at http://127.0.203.2:33163/ using document root <none> and password file <none>
I20260620 19:08:08.105377 4578 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:08.105422 4578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:08.105571 4578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:08.106630 4578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "d3d59e0420a64e30a7cbc6ead92f95e7"
format_stamp: "Formatted at 2026-06-20 19:08:08 on dist-test-slave-wd30"
server_key: "715dcff56ce741eeada41f52b6ef2571"
server_key_iv: "ebe4cc808fd610c4393bfb388fd39a6c"
server_key_version: "encryptionkey@0"
I20260620 19:08:08.106990 4578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "d3d59e0420a64e30a7cbc6ead92f95e7"
format_stamp: "Formatted at 2026-06-20 19:08:08 on dist-test-slave-wd30"
server_key: "715dcff56ce741eeada41f52b6ef2571"
server_key_iv: "ebe4cc808fd610c4393bfb388fd39a6c"
server_key_version: "encryptionkey@0"
I20260620 19:08:08.108456 4578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:08:08.109395 4593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:08.109549 4578 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:08:08.109751 4578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "d3d59e0420a64e30a7cbc6ead92f95e7"
format_stamp: "Formatted at 2026-06-20 19:08:08 on dist-test-slave-wd30"
server_key: "715dcff56ce741eeada41f52b6ef2571"
server_key_iv: "ebe4cc808fd610c4393bfb388fd39a6c"
server_key_version: "encryptionkey@0"
I20260620 19:08:08.109831 4578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:08.131665 4578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:08.134200 4578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:08.134341 4578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:08.134560 4578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:08.134851 4578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:08.134908 4578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:08.134932 4578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:08.134945 4578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:08.140955 4578 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:45401
I20260620 19:08:08.141004 4706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:45401 every 8 connection(s)
I20260620 19:08:08.141332 4578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:08:08.144811 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 4578
I20260620 19:08:08.144937 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:08:08.145124 812 external_mini_cluster.cc:1428] Setting key 5b77e5df46cd6bc4878e35789cc50f5b
Jun 20 19:08:08 dist-test-slave-wd30 krb5kdc[4337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:08.151947 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:08.142028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:47481 (local address 127.0.203.62:34477)
0620 19:08:08.142177 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:08.142179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:08.142388 (+ 209us) server_negotiation.cc:407] Connection header received
0620 19:08:08.143387 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:08.143388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:08.143400 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:08.143457 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:08.144104 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:08.144660 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:08.145949 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:08.146257 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:08.146336 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:08.148795 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:08.148796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:08.148797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:08.148805 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:08.150477 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:08.150867 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:08.150869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:08.150869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:08.150937 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:08.151221 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:08.151232 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:08.151232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:08.151398 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:08.151483 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:08.151742 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 19:08:08.151839 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 19:08:08.151968 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:08.142002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:34477 (local address 127.0.203.2:47481)
0620 19:08:08.142256 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:08.142294 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:08.143272 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:08.143538 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:08.143542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:08.143799 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:08.144006 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:08.144010 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:08.144775 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:08.144776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:08.145888 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:08.145890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:08.146371 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:08.146371 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:08.146487 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:08.147032 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:08.147041 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:08.148699 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:08.150566 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:08.150571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:08.150573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:08.150790 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:08.151013 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:08.151014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:08.151014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:08.151107 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:08.151473 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:08.151478 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:08.151595 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 19:08:08.151673 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 19:08:08.151801 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":93,"threads_started":1}
I20260620 19:08:08.152659 4707 heartbeater.cc:347] Connected to a master server at 127.0.203.62:34477
I20260620 19:08:08.152778 4707 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:08.152961 4707 heartbeater.cc:511] Master 127.0.203.62:34477 requested a full tablet report, sending...
I20260620 19:08:08.153342 4382 ts_manager.cc:194] Registered new tserver with Master: d3d59e0420a64e30a7cbc6ead92f95e7 (127.0.203.2:45401)
I20260620 19:08:08.153712 4382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:47481
WARNING: no policy specified for kudu/127.0.203.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
I20260620 19:08:08.171509 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:34477
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:08.254009 4712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:08.254276 4712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:08.254318 4712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:08.255621 4712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:08.255683 4712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:08.255728 4712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:08:08.257323 4712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:34477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:08.257601 4712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:08.257844 4712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:08.263185 4717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:08.263146 4718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:08.263345 4712 server_base.cc:1034] running on GCE node
W20260620 19:08:08.263152 4720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:08.263621 4712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:08.263901 4712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:08.265069 4712 hybrid_clock.cc:648] HybridClock initialized: now 1781982488265054 us; error 26 us; skew 500 ppm
Jun 20 19:08:08 dist-test-slave-wd30 krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:08.267729 4712 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:08:08.268200 4712 webserver.cc:460] Webserver started at http://127.0.203.3:44259/ using document root <none> and password file <none>
I20260620 19:08:08.268394 4712 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:08.268435 4712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:08.268523 4712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:08.269526 4712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "f985aa04df2647a58a658aeaedefa771"
format_stamp: "Formatted at 2026-06-20 19:08:08 on dist-test-slave-wd30"
server_key: "a0ce3374cc6b43f1288680d6d2913828"
server_key_iv: "188ceb38c463f0dabbef682cc6a57927"
server_key_version: "encryptionkey@0"
I20260620 19:08:08.269835 4712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "f985aa04df2647a58a658aeaedefa771"
format_stamp: "Formatted at 2026-06-20 19:08:08 on dist-test-slave-wd30"
server_key: "a0ce3374cc6b43f1288680d6d2913828"
server_key_iv: "188ceb38c463f0dabbef682cc6a57927"
server_key_version: "encryptionkey@0"
I20260620 19:08:08.271104 4712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:08:08.271871 4727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:08.272065 4712 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:08:08.272140 4712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "f985aa04df2647a58a658aeaedefa771"
format_stamp: "Formatted at 2026-06-20 19:08:08 on dist-test-slave-wd30"
server_key: "a0ce3374cc6b43f1288680d6d2913828"
server_key_iv: "188ceb38c463f0dabbef682cc6a57927"
server_key_version: "encryptionkey@0"
I20260620 19:08:08.272234 4712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:08.290738 4712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:08.293700 4712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:08.293866 4712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:08.294117 4712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:08.294441 4712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:08.294480 4712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:08.294504 4712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:08.294551 4712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:08.300503 4712 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:44029
I20260620 19:08:08.300577 4840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:44029 every 8 connection(s)
I20260620 19:08:08.300928 4712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:08:08.306545 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 4712
I20260620 19:08:08.306674 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:08:08.306911 812 external_mini_cluster.cc:1428] Setting key 8ae4195ee64169db02acaafcf8bb1202
Jun 20 19:08:08 dist-test-slave-wd30 krb5kdc[4337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:08.310941 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:08.301745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:40629 (local address 127.0.203.62:34477)
0620 19:08:08.301820 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:08.301822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:08.302061 (+ 239us) server_negotiation.cc:407] Connection header received
0620 19:08:08.303030 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:08.303031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:08.303043 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:08.303101 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:08.303821 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:08.304332 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:08.305436 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:08.305635 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:08.305692 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:08.308035 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:08.308036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:08.308036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:08.308044 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:08.309481 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:08.310022 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:08.310023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:08.310024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:08.310089 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:08.310387 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:08.310388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:08.310388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:08.310499 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:08.310554 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:08.310807 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 19:08:08.310875 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:08:08.311216 4843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:08.301714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:34477 (local address 127.0.203.3:40629)
0620 19:08:08.301934 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:08.301973 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:08.302918 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:08.303277 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:08.303281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:08.303541 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:08.303739 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:08.303743 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:08.304429 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:08.304430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:08.305341 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:08.305345 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:08.305710 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:08.305710 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:08.305816 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:08.306308 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:08.306316 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:08.307960 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:08.309554 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:08.309561 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:08.309563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:08.309946 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:08.310213 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:08.310214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:08.310214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:08.310320 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:08.310558 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:08.310562 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:08.310688 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 19:08:08.310774 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 19:08:08.310896 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":116,"threads_started":1}
I20260620 19:08:08.311707 4841 heartbeater.cc:347] Connected to a master server at 127.0.203.62:34477
I20260620 19:08:08.311794 4841 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:08.311992 4841 heartbeater.cc:511] Master 127.0.203.62:34477 requested a full tablet report, sending...
I20260620 19:08:08.312433 4382 ts_manager.cc:194] Registered new tserver with Master: f985aa04df2647a58a658aeaedefa771 (127.0.203.3:44029)
I20260620 19:08:08.312834 4382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:40629
I20260620 19:08:08.319167 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:08:08 dist-test-slave-wd30 krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:08:08 dist-test-slave-wd30 krb5kdc[4337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.1@KRBTEST.COM
I20260620 19:08:08.343223 4848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:08.334771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45192 (local address 127.0.203.1:40623)
0620 19:08:08.334962 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:08.334970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:08.334986 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:08:08.335013 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:08.335015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:08.335028 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:08.335103 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:08.335871 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:08.336625 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:08.337463 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:08.337796 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:08.337859 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:08.340189 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:08.340200 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:08.340200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:08.340207 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:08.341893 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:08.342374 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:08.342377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:08.342377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:08.342450 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:08.342628 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:08.342628 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:08.342629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:08.342783 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:08.342891 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:08.342966 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 19:08:08.343048 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":123,"threads_started":1}
Jun 20 19:08:08 dist-test-slave-wd30 krb5kdc[4337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:08.351629 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:08.344706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60442 (local address 127.0.203.62:34477)
0620 19:08:08.344770 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:08.344772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:08.344782 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:08:08.344806 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:08.344807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:08.344820 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:08.344887 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:08.345380 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:08.345961 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:08.346651 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:08.346843 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:08.346921 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:08.349180 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:08.349182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:08.349182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:08.349192 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:08.350399 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:08.350743 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:08.350744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:08.350745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:08.350803 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:08.351004 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:08.351004 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:08.351005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:08.351226 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:08.351345 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:08.351442 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 19:08:08.351523 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 19:08:08.351846 4382 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60442
I20260620 19:08:08.352206 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 4444
I20260620 19:08:08.357461 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 4578
I20260620 19:08:08.362076 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 4712
I20260620 19:08:08.366292 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 4350
2026-06-20T19:08:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1873 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:08:08 dist-test-slave-wd30 krb5kdc[4853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:08:08 dist-test-slave-wd30 krb5kdc[4853](info): set up 2 sockets
Jun 20 19:08:08 dist-test-slave-wd30 krb5kdc[4853](info): commencing operation
krb5kdc: starting...
W20260620 19:08:09.415072 812 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:08:09 dist-test-slave-wd30 krb5kdc[4853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:08:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:08:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:08:09.529677 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:38885
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:38885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:09.611996 4866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:09.612278 4866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:09.612322 4866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:09.613565 4866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:09.613647 4866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:09.613675 4866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:09.613708 4866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:09.613736 4866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:08:09.615324 4866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:38885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:38885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:09.615607 4866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:09.615867 4866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:09.620814 4871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:09.620887 4866 server_base.cc:1034] running on GCE node
W20260620 19:08:09.620831 4874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:09.620862 4872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:09.621243 4866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:09.621582 4866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:09.622730 4866 hybrid_clock.cc:648] HybridClock initialized: now 1781982489622710 us; error 36 us; skew 500 ppm
Jun 20 19:08:09 dist-test-slave-wd30 krb5kdc[4853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:09.628525 4866 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:08:09.628826 4866 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:09.628870 4866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:09.628960 4866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:09.629995 4866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "69e47746375845c1ab1b10150834170b"
format_stamp: "Formatted at 2026-06-20 19:08:09 on dist-test-slave-wd30"
server_key: "3468659c761cc056ce87eb80d4e9935f"
server_key_iv: "b6a015ee13660965bb7077c13e9b074e"
server_key_version: "encryptionkey@0"
I20260620 19:08:09.630298 4866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "69e47746375845c1ab1b10150834170b"
format_stamp: "Formatted at 2026-06-20 19:08:09 on dist-test-slave-wd30"
server_key: "3468659c761cc056ce87eb80d4e9935f"
server_key_iv: "b6a015ee13660965bb7077c13e9b074e"
server_key_version: "encryptionkey@0"
I20260620 19:08:09.631608 4866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 19:08:09.632315 4880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:09.632495 4866 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:09.632604 4866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "69e47746375845c1ab1b10150834170b"
format_stamp: "Formatted at 2026-06-20 19:08:09 on dist-test-slave-wd30"
server_key: "3468659c761cc056ce87eb80d4e9935f"
server_key_iv: "b6a015ee13660965bb7077c13e9b074e"
server_key_version: "encryptionkey@0"
I20260620 19:08:09.632681 4866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:09.645140 4866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:09.647431 4866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:09.647593 4866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:09.651553 4866 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:38885
I20260620 19:08:09.651582 4932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:38885 every 8 connection(s)
I20260620 19:08:09.651880 4866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:09.652839 4933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:09.654492 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 4866
I20260620 19:08:09.654610 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:08:09.654795 812 external_mini_cluster.cc:1428] Setting key 1e424fb65c36ea7ce4adc1aafec3b975
I20260620 19:08:09.655652 4933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b: Bootstrap starting.
I20260620 19:08:09.656304 4933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:09.656658 4933 log.cc:824] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:09.657423 4933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b: No bootstrap required, opened a new log
I20260620 19:08:09.659649 4933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e47746375845c1ab1b10150834170b" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 38885 } }
I20260620 19:08:09.659844 4933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:09.659932 4933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e47746375845c1ab1b10150834170b, State: Initialized, Role: FOLLOWER
I20260620 19:08:09.660074 4933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e47746375845c1ab1b10150834170b" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 38885 } }
I20260620 19:08:09.660169 4933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:09.660229 4933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:09.660280 4933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:09.660835 4933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e47746375845c1ab1b10150834170b" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 38885 } }
I20260620 19:08:09.660931 4933 leader_election.cc:302] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69e47746375845c1ab1b10150834170b; no voters:
I20260620 19:08:09.661142 4933 leader_election.cc:288] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:08:09.661168 4938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:09.661279 4938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b [term 1 LEADER]: Becoming Leader. State: Replica: 69e47746375845c1ab1b10150834170b, State: Running, Role: LEADER
I20260620 19:08:09.661355 4938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e47746375845c1ab1b10150834170b" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 38885 } }
I20260620 19:08:09.661422 4933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:09.661679 4938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69e47746375845c1ab1b10150834170b. Latest consensus state: current_term: 1 leader_uuid: "69e47746375845c1ab1b10150834170b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e47746375845c1ab1b10150834170b" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 38885 } } }
I20260620 19:08:09.661751 4938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:09.661691 4939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69e47746375845c1ab1b10150834170b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e47746375845c1ab1b10150834170b" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 38885 } } }
I20260620 19:08:09.661787 4939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b [sys.catalog]: This master's current role is: LEADER
Jun 20 19:08:09 dist-test-slave-wd30 krb5kdc[4853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:09.662101 4943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:08:09.662824 4943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:08:09.664440 4943 catalog_manager.cc:1347] Generated new cluster ID: 5b8df4e7ec134e67869baa9c61682311
I20260620 19:08:09.664495 4943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:08:09.666517 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:09.655349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52250 (local address 127.0.203.62:38885)
0620 19:08:09.655540 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:09.655556 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:09.655582 (+ 26us) server_negotiation.cc:407] Connection header received
0620 19:08:09.656055 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:09.656062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:09.656118 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:09.656233 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:09.657026 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:09.657992 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:09.659053 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:09.659526 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:09.659608 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:09.662382 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:09.662388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:09.662389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:09.662403 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:09.664465 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:09.664888 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:09.664892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:09.664892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:09.664996 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:09.665201 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:09.665202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:09.665202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:09.665512 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:09.665662 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:09.666144 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 19:08:09.666286 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":115,"threads_started":1}
I20260620 19:08:09.673839 4943 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:08:09.674396 4943 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:09.678460 4943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b: Generated new TSK 0
I20260620 19:08:09.678666 4943 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
I20260620 19:08:09.716255 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:38885
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:09.814586 4959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:09.814853 4959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:09.814900 4959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:09.816164 4959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:09.816246 4959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:09.816304 4959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:08:09.817905 4959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:38885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:09.818202 4959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:09.818463 4959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:09.823625 4965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:09.823648 4964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:09.823616 4967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:09.823624 4959 server_base.cc:1034] running on GCE node
I20260620 19:08:09.824028 4959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:09.824237 4959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:09.825376 4959 hybrid_clock.cc:648] HybridClock initialized: now 1781982489825353 us; error 41 us; skew 500 ppm
Jun 20 19:08:09 dist-test-slave-wd30 krb5kdc[4853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:09.828083 4959 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:08:09.828383 4959 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:09.828426 4959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:09.828519 4959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:09.829530 4959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "6131942299944d639b42ff5965b527ff"
format_stamp: "Formatted at 2026-06-20 19:08:09 on dist-test-slave-wd30"
server_key: "5a4563a9d5b5711eeeffde97724ebd22"
server_key_iv: "2aff3aa365828fe1850e264723394fb2"
server_key_version: "encryptionkey@0"
I20260620 19:08:09.829833 4959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "6131942299944d639b42ff5965b527ff"
format_stamp: "Formatted at 2026-06-20 19:08:09 on dist-test-slave-wd30"
server_key: "5a4563a9d5b5711eeeffde97724ebd22"
server_key_iv: "2aff3aa365828fe1850e264723394fb2"
server_key_version: "encryptionkey@0"
I20260620 19:08:09.831085 4959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:08:09.831876 4973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:09.832042 4959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:09.832154 4959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "6131942299944d639b42ff5965b527ff"
format_stamp: "Formatted at 2026-06-20 19:08:09 on dist-test-slave-wd30"
server_key: "5a4563a9d5b5711eeeffde97724ebd22"
server_key_iv: "2aff3aa365828fe1850e264723394fb2"
server_key_version: "encryptionkey@0"
I20260620 19:08:09.832231 4959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:09.841703 4959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:09.844146 4959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:09.844318 4959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:09.844556 4959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:09.844866 4959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:09.844919 4959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:09.844962 4959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:09.845016 4959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:09.850692 4959 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:37115
I20260620 19:08:09.850775 5086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:37115 every 8 connection(s)
I20260620 19:08:09.851063 4959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:08:09.852129 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 4959
I20260620 19:08:09.852231 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:08:09.852447 812 external_mini_cluster.cc:1428] Setting key 706f4983ff9f5b34c4d5f4bd58649708
Jun 20 19:08:09 dist-test-slave-wd30 krb5kdc[4853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.203.2@KRBTEST.COM; defaulting to no policy
I20260620 19:08:09.862280 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:09.851861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:33521 (local address 127.0.203.62:38885)
0620 19:08:09.851921 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:09.851923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:09.852190 (+ 267us) server_negotiation.cc:407] Connection header received
0620 19:08:09.853001 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:09.853002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:09.853010 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:09.853091 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:09.854451 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:09.855307 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:09.856323 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:09.856695 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:09.856774 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:09.859262 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:09.859264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:09.859264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:09.859273 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:09.860870 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:09.861268 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:09.861269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:09.861269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:09.861336 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:09.861670 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:09.861671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:09.861672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:09.861809 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:09.861908 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:09.862143 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 19:08:09.862212 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:08:09.862537 5089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:09.851846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:38885 (local address 127.0.203.1:33521)
0620 19:08:09.852069 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:09.852106 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:09.852712 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:09.853199 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:09.853204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:09.853632 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:09.854342 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:09.854349 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:09.855426 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:09.855427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:09.856230 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:09.856232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:09.856792 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:09.856793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:09.856917 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:09.857499 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:09.857507 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:09.859115 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:09.860958 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:09.860964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:09.860966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:09.861188 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:09.861461 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:09.861462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:09.861462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:09.861596 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:09.861896 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:09.861900 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:09.862029 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 19:08:09.862113 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:08:09.862235 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":144,"threads_started":1}
I20260620 19:08:09.863073 5087 heartbeater.cc:347] Connected to a master server at 127.0.203.62:38885
I20260620 19:08:09.863201 5087 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:09.863335 5087 heartbeater.cc:511] Master 127.0.203.62:38885 requested a full tablet report, sending...
I20260620 19:08:09.863852 4897 ts_manager.cc:194] Registered new tserver with Master: 6131942299944d639b42ff5965b527ff (127.0.203.1:37115)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.2@KRBTEST.COM" created.
I20260620 19:08:09.864526 4897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:33521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
I20260620 19:08:09.878294 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:38885
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:09.959969 5092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:09.960184 5092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:09.960209 5092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:09.961416 5092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:09.961460 5092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:09.961488 5092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:08:09.963236 5092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:38885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:09.963487 5092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:09.963723 5092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:08:09.969022 5092 server_base.cc:1034] running on GCE node
W20260620 19:08:09.968988 5097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:09.968962 5100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:09.968971 5098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:09.969336 5092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:09.969655 5092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:09.970808 5092 hybrid_clock.cc:648] HybridClock initialized: now 1781982489970780 us; error 41 us; skew 500 ppm
Jun 20 19:08:09 dist-test-slave-wd30 krb5kdc[4853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:09.973395 5092 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:08:09.973769 5092 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:09.973819 5092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:09.973920 5092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:09.974948 5092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "7d4b117b6e784f729778816fce04c6f5"
format_stamp: "Formatted at 2026-06-20 19:08:09 on dist-test-slave-wd30"
server_key: "3c6d8113f9b058076fb000f5df24cfe8"
server_key_iv: "4b9e0c7c502d41e99f483b6719688a45"
server_key_version: "encryptionkey@0"
I20260620 19:08:09.975330 5092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "7d4b117b6e784f729778816fce04c6f5"
format_stamp: "Formatted at 2026-06-20 19:08:09 on dist-test-slave-wd30"
server_key: "3c6d8113f9b058076fb000f5df24cfe8"
server_key_iv: "4b9e0c7c502d41e99f483b6719688a45"
server_key_version: "encryptionkey@0"
I20260620 19:08:09.976563 5092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:08:09.977245 5106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:09.977419 5092 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:08:09.977528 5092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "7d4b117b6e784f729778816fce04c6f5"
format_stamp: "Formatted at 2026-06-20 19:08:09 on dist-test-slave-wd30"
server_key: "3c6d8113f9b058076fb000f5df24cfe8"
server_key_iv: "4b9e0c7c502d41e99f483b6719688a45"
server_key_version: "encryptionkey@0"
I20260620 19:08:09.977602 5092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:10.000088 5092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:10.002175 5092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:10.002355 5092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:10.002619 5092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:10.003006 5092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:10.003062 5092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.003105 5092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:10.003152 5092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.009104 5092 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:34573
I20260620 19:08:10.009142 5219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:34573 every 8 connection(s)
I20260620 19:08:10.009418 5092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:08:10.013058 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 5092
I20260620 19:08:10.013151 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:08:10.013399 812 external_mini_cluster.cc:1428] Setting key 1647ab39d39a722d459a2adff50ee5c2
Jun 20 19:08:10 dist-test-slave-wd30 krb5kdc[4853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:10.020983 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.010173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:60167 (local address 127.0.203.62:38885)
0620 19:08:10.010234 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.010236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.010424 (+ 188us) server_negotiation.cc:407] Connection header received
0620 19:08:10.010903 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.010904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.010914 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.010975 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:10.012181 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.012907 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.015228 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.015419 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.015519 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.017742 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:10.017745 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:10.017745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:10.017755 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:10.019346 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:10.019893 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:10.019895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:10.019895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:10.019960 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:10.020271 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:10.020272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:10.020272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:10.020390 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:10.020446 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.020810 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.020914 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:08:10.021234 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.010124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:38885 (local address 127.0.203.2:60167)
0620 19:08:10.010296 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.010338 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.010812 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.011057 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.011060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.011331 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:10.012059 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.012068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.012999 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.012999 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.014086 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.014089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.015530 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.015531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.015664 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.016217 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:10.016225 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:10.017640 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:10.019429 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:10.019439 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:10.019441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:10.019768 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:10.020052 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:10.020053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:10.020053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:10.020200 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:10.020465 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:10.020471 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:10.020652 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.020783 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.020956 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260620 19:08:10.021796 5220 heartbeater.cc:347] Connected to a master server at 127.0.203.62:38885
I20260620 19:08:10.021934 5220 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:10.022094 5220 heartbeater.cc:511] Master 127.0.203.62:38885 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.203.3@KRBTEST.COM; defaulting to no policy
I20260620 19:08:10.022517 4897 ts_manager.cc:194] Registered new tserver with Master: 7d4b117b6e784f729778816fce04c6f5 (127.0.203.2:34573)
I20260620 19:08:10.022902 4897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:60167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
I20260620 19:08:10.038344 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:38885
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:10.119778 5225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:10.120051 5225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:10.120090 5225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:10.121268 5225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:10.121347 5225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:10.121393 5225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:08:10.122956 5225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:38885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:10.123265 5225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:10.123519 5225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:08:10.128696 5225 server_base.cc:1034] running on GCE node
W20260620 19:08:10.128648 5230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:10.128648 5233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:10.128677 5231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:10.129107 5225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:10.129376 5225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:10.130533 5225 hybrid_clock.cc:648] HybridClock initialized: now 1781982490130525 us; error 32 us; skew 500 ppm
Jun 20 19:08:10 dist-test-slave-wd30 krb5kdc[4853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:10.133083 5225 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:08:10.133435 5225 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:10.133507 5225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:10.133658 5225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:10.134737 5225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "f142f12664ce45f9bd99683c4138ab2a"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-wd30"
server_key: "891ae4b4b94dcaa7e57b0dc727af812a"
server_key_iv: "ca41b42295be85c28862937a03586d72"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.135095 5225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "f142f12664ce45f9bd99683c4138ab2a"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-wd30"
server_key: "891ae4b4b94dcaa7e57b0dc727af812a"
server_key_iv: "ca41b42295be85c28862937a03586d72"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.136531 5225 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:10.137328 5239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.137566 5225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:10.137652 5225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "f142f12664ce45f9bd99683c4138ab2a"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-wd30"
server_key: "891ae4b4b94dcaa7e57b0dc727af812a"
server_key_iv: "ca41b42295be85c28862937a03586d72"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.137748 5225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:10.153831 5225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:10.156152 5225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:10.156306 5225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:10.156558 5225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:10.156949 5225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:10.157013 5225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.157058 5225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:10.157080 5225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.163293 5225 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:45255
I20260620 19:08:10.163341 5352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:45255 every 8 connection(s)
I20260620 19:08:10.163618 5225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:08:10 dist-test-slave-wd30 krb5kdc[4853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:10.172865 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.164306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:46257 (local address 127.0.203.62:38885)
0620 19:08:10.164366 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.164368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.164582 (+ 214us) server_negotiation.cc:407] Connection header received
0620 19:08:10.165121 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.165122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.165135 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.165191 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:10.166184 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.166689 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.167797 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.167992 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.168087 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.170218 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:10.170219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:10.170220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:10.170229 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:10.171492 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:10.171898 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:10.171899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:10.171900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:10.171959 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:10.172203 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:10.172204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:10.172205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:10.172322 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:10.172400 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.172687 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.172770 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:08:10.173060 5355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.164304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:38885 (local address 127.0.203.3:46257)
0620 19:08:10.164452 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.164488 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.165016 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.165263 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.165268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.165556 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:10.166052 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.166057 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.166787 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.166788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.167690 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.167693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.168095 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.168096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.168228 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.168721 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:10.168730 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:10.170122 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:10.171584 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:10.171591 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:10.171593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:10.171809 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:10.172030 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:10.172031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:10.172031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:10.172138 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:10.172389 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:10.172395 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:10.172551 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.172664 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.172822 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
I20260620 19:08:10.173151 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 5225
I20260620 19:08:10.173242 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:08:10.173449 812 external_mini_cluster.cc:1428] Setting key a330ce9e9367e08dcf5127ed0d85ab00
I20260620 19:08:10.173599 5353 heartbeater.cc:347] Connected to a master server at 127.0.203.62:38885
I20260620 19:08:10.173684 5353 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:10.173853 5353 heartbeater.cc:511] Master 127.0.203.62:38885 requested a full tablet report, sending...
I20260620 19:08:10.174252 4896 ts_manager.cc:194] Registered new tserver with Master: f142f12664ce45f9bd99683c4138ab2a (127.0.203.3:45255)
I20260620 19:08:10.174666 4896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:46257
I20260620 19:08:10.185473 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:08:10.193284 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.186953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.0.203.62:38885)
0620 19:08:10.187045 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.187048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.187155 (+ 107us) server_negotiation.cc:407] Connection header received
0620 19:08:10.187249 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.187250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.187262 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.187338 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:10.187921 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.188433 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.189151 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.189356 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.189432 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.190606 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:10.190608 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:10.190608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:10.190625 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:10.192004 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:10.192362 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:10.192363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:10.192364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:10.192423 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:10.192624 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:10.192625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:10.192626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:10.192794 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:10.192907 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.193109 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.193187 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 19:08:10.195070 4896 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:08:10.196667 4896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:08:10.203240 5368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.199040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:34573 (local address 127.0.0.1:48878)
0620 19:08:10.199372 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.199382 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.199423 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.199918 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.199918 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.199921 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:10.199946 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.199947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.201168 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.201169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.201983 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.201985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.202918 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.202919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.203010 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.203023 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.203092 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.203162 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":17,"thread_start_us":121,"threads_started":1}
I20260620 19:08:10.203310 5367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.199198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48878 (local address 127.0.203.2:34573)
0620 19:08:10.199702 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.199706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.199714 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:08:10.199749 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.199751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.199766 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.199835 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:10.200012 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.201063 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.202086 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.202756 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.202828 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.202885 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.203169 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.203221 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":78,"threads_started":1}
I20260620 19:08:10.203601 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.199137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.0.203.1:37115)
0620 19:08:10.199326 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.199332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.199344 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:08:10.199434 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.199436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.199469 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.199563 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:10.199774 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.200902 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.202417 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.203114 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.203203 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.203261 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.203480 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.203528 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":74,"threads_started":1}
I20260620 19:08:10.204010 5154 tablet_service.cc:1467] Processing CreateTablet for tablet 6819b0789a9344aab2f633dcb0b21eba (DEFAULT_TABLE table=test-table [id=58e6fd1457af4586816e4512b9d215dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:10.204355 5154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6819b0789a9344aab2f633dcb0b21eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:10.204792 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.199263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:45255 (local address 127.0.0.1:36362)
0620 19:08:10.199446 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.199454 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.199489 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.199875 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.199876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.199878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:10.199931 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.199933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.202367 (+ 2434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.202368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.203801 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.203803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.204561 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.204562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.204622 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.204626 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.204684 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.204722 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260620 19:08:10.204810 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.199461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36362 (local address 127.0.203.3:45255)
0620 19:08:10.199638 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.199642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.199654 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:08:10.199691 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.199693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.199709 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.199788 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:10.201186 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.202261 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.203935 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.204489 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.204560 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.204614 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.204697 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.204732 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":80,"threads_started":1}
I20260620 19:08:10.205574 5287 tablet_service.cc:1467] Processing CreateTablet for tablet 6819b0789a9344aab2f633dcb0b21eba (DEFAULT_TABLE table=test-table [id=58e6fd1457af4586816e4512b9d215dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:10.205896 5287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6819b0789a9344aab2f633dcb0b21eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:10.207698 5370 tablet_bootstrap.cc:492] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5: Bootstrap starting.
I20260620 19:08:10.207955 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.199005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:37115 (local address 127.0.0.1:49226)
0620 19:08:10.199215 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.199258 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.199324 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.199571 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.199578 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.199580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:10.199632 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.199633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.201047 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.201048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.202319 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.202321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.203295 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.203296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.203362 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.203367 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.203454 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.203494 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":22,"thread_start_us":67,"threads_started":1}
I20260620 19:08:10.208346 5370 tablet_bootstrap.cc:654] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:10.208657 5021 tablet_service.cc:1467] Processing CreateTablet for tablet 6819b0789a9344aab2f633dcb0b21eba (DEFAULT_TABLE table=test-table [id=58e6fd1457af4586816e4512b9d215dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:10.208889 5370 log.cc:824] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:10.208973 5021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6819b0789a9344aab2f633dcb0b21eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:10.209436 5371 tablet_bootstrap.cc:492] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a: Bootstrap starting.
I20260620 19:08:10.209757 5370 tablet_bootstrap.cc:492] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5: No bootstrap required, opened a new log
I20260620 19:08:10.209825 5370 ts_tablet_manager.cc:1397] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:08:10.210072 5371 tablet_bootstrap.cc:654] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:10.210636 5371 log.cc:824] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:10.211464 5371 tablet_bootstrap.cc:492] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a: No bootstrap required, opened a new log
I20260620 19:08:10.211534 5371 ts_tablet_manager.cc:1397] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:08:10.211907 5374 tablet_bootstrap.cc:492] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff: Bootstrap starting.
I20260620 19:08:10.211977 5370 raft_consensus.cc:348] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6131942299944d639b42ff5965b527ff" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37115 } } peers { permanent_uuid: "7d4b117b6e784f729778816fce04c6f5" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 34573 } } peers { permanent_uuid: "f142f12664ce45f9bd99683c4138ab2a" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 45255 } }
I20260620 19:08:10.212150 5370 raft_consensus.cc:374] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:10.212183 5370 raft_consensus.cc:729] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4b117b6e784f729778816fce04c6f5, State: Initialized, Role: FOLLOWER
I20260620 19:08:10.212304 5370 consensus_queue.cc:260] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6131942299944d639b42ff5965b527ff" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37115 } } peers { permanent_uuid: "7d4b117b6e784f729778816fce04c6f5" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 34573 } } peers { permanent_uuid: "f142f12664ce45f9bd99683c4138ab2a" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 45255 } }
I20260620 19:08:10.212598 5370 ts_tablet_manager.cc:1428] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:08:10.212617 5220 heartbeater.cc:503] Master 127.0.203.62:38885 was elected leader, sending a full tablet report...
I20260620 19:08:10.212687 5374 tablet_bootstrap.cc:654] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:10.213287 5374 log.cc:824] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:10.213565 5371 raft_consensus.cc:348] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6131942299944d639b42ff5965b527ff" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37115 } } peers { permanent_uuid: "7d4b117b6e784f729778816fce04c6f5" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 34573 } } peers { permanent_uuid: "f142f12664ce45f9bd99683c4138ab2a" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 45255 } }
I20260620 19:08:10.213721 5371 raft_consensus.cc:374] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:10.213872 5371 raft_consensus.cc:729] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f142f12664ce45f9bd99683c4138ab2a, State: Initialized, Role: FOLLOWER
I20260620 19:08:10.214027 5371 consensus_queue.cc:260] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6131942299944d639b42ff5965b527ff" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37115 } } peers { permanent_uuid: "7d4b117b6e784f729778816fce04c6f5" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 34573 } } peers { permanent_uuid: "f142f12664ce45f9bd99683c4138ab2a" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 45255 } }
I20260620 19:08:10.214205 5374 tablet_bootstrap.cc:492] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff: No bootstrap required, opened a new log
I20260620 19:08:10.214274 5374 ts_tablet_manager.cc:1397] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:08:10.214301 5371 ts_tablet_manager.cc:1428] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:08:10.214299 5353 heartbeater.cc:503] Master 127.0.203.62:38885 was elected leader, sending a full tablet report...
I20260620 19:08:10.215672 5374 raft_consensus.cc:348] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6131942299944d639b42ff5965b527ff" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37115 } } peers { permanent_uuid: "7d4b117b6e784f729778816fce04c6f5" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 34573 } } peers { permanent_uuid: "f142f12664ce45f9bd99683c4138ab2a" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 45255 } }
I20260620 19:08:10.215812 5374 raft_consensus.cc:374] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:10.215855 5374 raft_consensus.cc:729] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6131942299944d639b42ff5965b527ff, State: Initialized, Role: FOLLOWER
I20260620 19:08:10.215962 5374 consensus_queue.cc:260] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6131942299944d639b42ff5965b527ff" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37115 } } peers { permanent_uuid: "7d4b117b6e784f729778816fce04c6f5" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 34573 } } peers { permanent_uuid: "f142f12664ce45f9bd99683c4138ab2a" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 45255 } }
I20260620 19:08:10.216229 5374 ts_tablet_manager.cc:1428] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:08:10.216259 5087 heartbeater.cc:503] Master 127.0.203.62:38885 was elected leader, sending a full tablet report...
W20260620 19:08:10.259968 5221 tablet.cc:2369] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:08:10.292476 5378 raft_consensus.cc:482] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:08:10.292621 5378 raft_consensus.cc:504] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6131942299944d639b42ff5965b527ff" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37115 } } peers { permanent_uuid: "7d4b117b6e784f729778816fce04c6f5" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 34573 } } peers { permanent_uuid: "f142f12664ce45f9bd99683c4138ab2a" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 45255 } }
I20260620 19:08:10.292943 5378 leader_election.cc:288] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d4b117b6e784f729778816fce04c6f5 (127.0.203.2:34573), f142f12664ce45f9bd99683c4138ab2a (127.0.203.3:45255)
I20260620 19:08:10.296248 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.293405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:44891 (local address 127.0.203.3:45255)
0620 19:08:10.293450 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.293452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.293641 (+ 189us) server_negotiation.cc:407] Connection header received
0620 19:08:10.293670 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.293671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.293690 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.293759 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:10.293913 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.294522 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.295512 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.295888 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.295944 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.295953 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.296131 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.296180 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:08:10.296334 5367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.293389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:36595 (local address 127.0.203.2:34573)
0620 19:08:10.293474 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.293476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.293486 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:08:10.293521 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.293522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.293532 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.293594 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:10.293817 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.294468 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.295545 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.295937 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.296009 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.296018 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.296218 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.296260 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:08:10.296365 5089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.293226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:34573 (local address 127.0.203.1:36595)
0620 19:08:10.293366 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.293379 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.293431 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.293630 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.293631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.293632 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:10.293715 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.293733 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.294553 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.294555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.295424 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.295427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.296031 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.296032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.296130 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.296134 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.296190 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.296225 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":14}
I20260620 19:08:10.296257 5379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.293355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:45255 (local address 127.0.203.1:44891)
0620 19:08:10.293544 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.293552 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.293620 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.293768 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.293769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.293771 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:10.293831 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.293832 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.294610 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.294611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.295423 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:10.295426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.295970 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.295971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:10.296045 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.296050 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.296102 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.296138 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":116,"threads_started":1}
I20260620 19:08:10.296782 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6819b0789a9344aab2f633dcb0b21eba" candidate_uuid: "6131942299944d639b42ff5965b527ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4b117b6e784f729778816fce04c6f5" is_pre_election: true
I20260620 19:08:10.296850 5307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6819b0789a9344aab2f633dcb0b21eba" candidate_uuid: "6131942299944d639b42ff5965b527ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f142f12664ce45f9bd99683c4138ab2a" is_pre_election: true
I20260620 19:08:10.296970 5307 raft_consensus.cc:2454] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6131942299944d639b42ff5965b527ff in term 0.
I20260620 19:08:10.296970 5174 raft_consensus.cc:2454] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6131942299944d639b42ff5965b527ff in term 0.
I20260620 19:08:10.297189 4974 leader_election.cc:302] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6131942299944d639b42ff5965b527ff, f142f12664ce45f9bd99683c4138ab2a; no voters:
I20260620 19:08:10.297314 5378 raft_consensus.cc:2781] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:08:10.297365 5378 raft_consensus.cc:482] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:08:10.297405 5378 raft_consensus.cc:3037] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:10.298102 5378 raft_consensus.cc:504] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6131942299944d639b42ff5965b527ff" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37115 } } peers { permanent_uuid: "7d4b117b6e784f729778816fce04c6f5" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 34573 } } peers { permanent_uuid: "f142f12664ce45f9bd99683c4138ab2a" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 45255 } }
I20260620 19:08:10.298278 5378 leader_election.cc:288] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff [CANDIDATE]: Term 1 election: Requested vote from peers 7d4b117b6e784f729778816fce04c6f5 (127.0.203.2:34573), f142f12664ce45f9bd99683c4138ab2a (127.0.203.3:45255)
I20260620 19:08:10.298533 5307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6819b0789a9344aab2f633dcb0b21eba" candidate_uuid: "6131942299944d639b42ff5965b527ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f142f12664ce45f9bd99683c4138ab2a"
I20260620 19:08:10.298555 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6819b0789a9344aab2f633dcb0b21eba" candidate_uuid: "6131942299944d639b42ff5965b527ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4b117b6e784f729778816fce04c6f5"
I20260620 19:08:10.298638 5307 raft_consensus.cc:3037] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:10.298646 5174 raft_consensus.cc:3037] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:10.299428 5174 raft_consensus.cc:2454] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6131942299944d639b42ff5965b527ff in term 1.
I20260620 19:08:10.299573 5307 raft_consensus.cc:2454] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6131942299944d639b42ff5965b527ff in term 1.
I20260620 19:08:10.299624 4974 leader_election.cc:302] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6131942299944d639b42ff5965b527ff, 7d4b117b6e784f729778816fce04c6f5; no voters:
I20260620 19:08:10.299713 5378 raft_consensus.cc:2781] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:10.299873 5378 raft_consensus.cc:686] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff [term 1 LEADER]: Becoming Leader. State: Replica: 6131942299944d639b42ff5965b527ff, State: Running, Role: LEADER
I20260620 19:08:10.299976 5378 consensus_queue.cc:237] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6131942299944d639b42ff5965b527ff" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37115 } } peers { permanent_uuid: "7d4b117b6e784f729778816fce04c6f5" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 34573 } } peers { permanent_uuid: "f142f12664ce45f9bd99683c4138ab2a" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 45255 } }
I20260620 19:08:10.301023 4897 catalog_manager.cc:5495] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 6131942299944d639b42ff5965b527ff (127.0.203.1). New cstate: current_term: 1 leader_uuid: "6131942299944d639b42ff5965b527ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6131942299944d639b42ff5965b527ff" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d4b117b6e784f729778816fce04c6f5" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 34573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f142f12664ce45f9bd99683c4138ab2a" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 45255 } health_report { overall_health: UNKNOWN } } }
I20260620 19:08:10.321828 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.319407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49242 (local address 127.0.203.1:37115)
0620 19:08:10.319475 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.319477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.319486 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:08:10.319529 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.319530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.319567 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.319646 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:08:10.319797 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.320265 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.321019 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:10.321217 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:10.321295 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:10.321473 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:08:10.321580 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:08:10.321677 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.321696 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.321716 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:08:10.323791 5174 raft_consensus.cc:1261] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5 [term 1 FOLLOWER]: Refusing update from remote peer 6131942299944d639b42ff5965b527ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:10.323791 5307 raft_consensus.cc:1261] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a [term 1 FOLLOWER]: Refusing update from remote peer 6131942299944d639b42ff5965b527ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:10.324170 5380 consensus_queue.cc:1035] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4b117b6e784f729778816fce04c6f5" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 34573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:10.324326 5378 consensus_queue.cc:1035] T 6819b0789a9344aab2f633dcb0b21eba P 6131942299944d639b42ff5965b527ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "f142f12664ce45f9bd99683c4138ab2a" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 45255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:10.329737 4897 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:10.329838 4897 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:10.330603 4897 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69e47746375845c1ab1b10150834170b: Sending DeleteTablet for 3 replicas of tablet 6819b0789a9344aab2f633dcb0b21eba
I20260620 19:08:10.330912 5021 tablet_service.cc:1514] Processing DeleteTablet for tablet 6819b0789a9344aab2f633dcb0b21eba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:10 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:49226
I20260620 19:08:10.331009 5154 tablet_service.cc:1514] Processing DeleteTablet for tablet 6819b0789a9344aab2f633dcb0b21eba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:10 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:48878
I20260620 19:08:10.331027 5287 tablet_service.cc:1514] Processing DeleteTablet for tablet 6819b0789a9344aab2f633dcb0b21eba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:10 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:36362
I20260620 19:08:10.331286 5395 tablet_replica.cc:330] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a: stopping tablet replica
I20260620 19:08:10.331425 5395 raft_consensus.cc:2229] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:10.331506 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 4959
I20260620 19:08:10.332062 5395 raft_consensus.cc:2258] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:10.332319 5394 tablet_replica.cc:330] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5: stopping tablet replica
I20260620 19:08:10.332412 5394 raft_consensus.cc:2229] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:10.332511 5394 raft_consensus.cc:2258] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:10.333345 5394 ts_tablet_manager.cc:1905] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:10.334049 5395 ts_tablet_manager.cc:1905] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:10.334936 5394 ts_tablet_manager.cc:1918] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:10.335013 5394 log.cc:1196] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/6819b0789a9344aab2f633dcb0b21eba
I20260620 19:08:10.335327 5394 ts_tablet_manager.cc:1939] T 6819b0789a9344aab2f633dcb0b21eba P 7d4b117b6e784f729778816fce04c6f5: Deleting consensus metadata
I20260620 19:08:10.335753 4881 catalog_manager.cc:4841] TS 7d4b117b6e784f729778816fce04c6f5 (127.0.203.2:34573): tablet 6819b0789a9344aab2f633dcb0b21eba (table test-table [id=58e6fd1457af4586816e4512b9d215dc]) successfully deleted
I20260620 19:08:10.335799 5395 ts_tablet_manager.cc:1918] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:10.335845 5395 log.cc:1196] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/6819b0789a9344aab2f633dcb0b21eba
I20260620 19:08:10.336108 5395 ts_tablet_manager.cc:1939] T 6819b0789a9344aab2f633dcb0b21eba P f142f12664ce45f9bd99683c4138ab2a: Deleting consensus metadata
I20260620 19:08:10.336496 4881 catalog_manager.cc:4841] TS f142f12664ce45f9bd99683c4138ab2a (127.0.203.3:45255): tablet 6819b0789a9344aab2f633dcb0b21eba (table test-table [id=58e6fd1457af4586816e4512b9d215dc]) successfully deleted
W20260620 19:08:10.337306 4882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.1:37115 (error 108)
I20260620 19:08:10.337505 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 5092
I20260620 19:08:10.338070 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.337667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:37115 (local address 127.0.0.1:49244)
0620 19:08:10.337991 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.338030 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.1:37115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305}
W20260620 19:08:10.338202 4882 catalog_manager.cc:4584] TS 6131942299944d639b42ff5965b527ff (127.0.203.1:37115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6819b0789a9344aab2f633dcb0b21eba: Network error: Client connection negotiation failed: client connection to 127.0.203.1:37115: connect: Connection refused (error 111)
I20260620 19:08:10.342463 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 5225
I20260620 19:08:10.346817 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 4866
2026-06-20T19:08:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1980 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T19:08:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:08:10Z Disabled control of system clock
I20260620 19:08:10.374341 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:35101
--webserver_interface=127.0.203.62
--webserver_port=0
--builtin_ntp_servers=127.0.203.20:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:35101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 19:08:10.455793 5400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:10.456013 5400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:10.456035 5400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:10.457197 5400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:10.457240 5400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:10.457253 5400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:10.457266 5400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:10.457322 5400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:08:10.458781 5400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:35101
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:35101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:10.459048 5400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:10.459378 5400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:10.464088 5406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:10.464076 5405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:10.464072 5408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:10.464398 5400 server_base.cc:1034] running on GCE node
I20260620 19:08:10.464668 5400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:10.464953 5400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:10.466109 5400 hybrid_clock.cc:648] HybridClock initialized: now 1781982490466095 us; error 28 us; skew 500 ppm
I20260620 19:08:10.467427 5400 webserver.cc:460] Webserver started at http://127.0.203.62:43283/ using document root <none> and password file <none>
I20260620 19:08:10.467689 5400 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:10.467736 5400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:10.467885 5400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:10.469050 5400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "06bd1d99e67a481db3c58620a5022d9f"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-wd30"
server_key: "b9fdd7fa8152334f42f4f4273f29d2df"
server_key_iv: "d3747b10954240eb29e92738cf1c1556"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.469436 5400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "06bd1d99e67a481db3c58620a5022d9f"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-wd30"
server_key: "b9fdd7fa8152334f42f4f4273f29d2df"
server_key_iv: "d3747b10954240eb29e92738cf1c1556"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.470739 5400 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:08:10.471519 5414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.471704 5400 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.471794 5400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "06bd1d99e67a481db3c58620a5022d9f"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-wd30"
server_key: "b9fdd7fa8152334f42f4f4273f29d2df"
server_key_iv: "d3747b10954240eb29e92738cf1c1556"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.471868 5400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:10.475186 5400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:10.475459 5400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:10.475603 5400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:10.479638 5400 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:35101
I20260620 19:08:10.479699 5466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:35101 every 8 connection(s)
I20260620 19:08:10.480006 5400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:10.480639 5467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:10.483412 5467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f: Bootstrap starting.
I20260620 19:08:10.483973 5467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:10.484267 5467 log.cc:824] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:10.484956 5467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f: No bootstrap required, opened a new log
I20260620 19:08:10.486492 5467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06bd1d99e67a481db3c58620a5022d9f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35101 } }
I20260620 19:08:10.486655 5467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:10.486703 5467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06bd1d99e67a481db3c58620a5022d9f, State: Initialized, Role: FOLLOWER
I20260620 19:08:10.486853 5467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06bd1d99e67a481db3c58620a5022d9f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35101 } }
I20260620 19:08:10.486931 5467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:10.486974 5467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:10.487020 5467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:10.487574 5467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06bd1d99e67a481db3c58620a5022d9f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35101 } }
I20260620 19:08:10.487689 5467 leader_election.cc:302] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06bd1d99e67a481db3c58620a5022d9f; no voters:
I20260620 19:08:10.487869 5467 leader_election.cc:288] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:08:10.487919 5470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:10.488147 5470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f [term 1 LEADER]: Becoming Leader. State: Replica: 06bd1d99e67a481db3c58620a5022d9f, State: Running, Role: LEADER
I20260620 19:08:10.488144 5467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:10.488276 5470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06bd1d99e67a481db3c58620a5022d9f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35101 } }
I20260620 19:08:10.488677 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 5400
I20260620 19:08:10.488772 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:08:10.488963 812 external_mini_cluster.cc:1428] Setting key 93d7fdd0ab78196568dede0d1503f8f5
I20260620 19:08:10.489519 5472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06bd1d99e67a481db3c58620a5022d9f. Latest consensus state: current_term: 1 leader_uuid: "06bd1d99e67a481db3c58620a5022d9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06bd1d99e67a481db3c58620a5022d9f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35101 } } }
I20260620 19:08:10.489610 5472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:10.489810 5487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:08:10.489867 5471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06bd1d99e67a481db3c58620a5022d9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06bd1d99e67a481db3c58620a5022d9f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35101 } } }
I20260620 19:08:10.490100 5471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:10.490501 5487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:08:10.491601 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.489452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.0.203.62:35101)
0620 19:08:10.490028 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.490038 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.490073 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:08:10.490188 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.490196 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.490243 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.490395 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:10.491094 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:10.491097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:10.491098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:10.491102 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:10.491171 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:10.491185 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:10.491254 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.491332 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.491402 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":126,"threads_started":1}
I20260620 19:08:10.492322 5487 catalog_manager.cc:1347] Generated new cluster ID: ab37625f77494d229f2f525669bcfef7
I20260620 19:08:10.492383 5487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:08:10.497074 5487 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:08:10.497244 5487 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:10.503448 5487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f: Generated new TSK 0
I20260620 19:08:10.503646 5487 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:08:10.508082 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:35101
--builtin_ntp_servers=127.0.203.20:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 19:08:10.591714 5491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:10.591948 5491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:10.591988 5491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:10.593266 5491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:10.593324 5491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:10.593369 5491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:08:10.594925 5491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:35101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:10.595220 5491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:10.595526 5491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:10.600613 5496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:10.600638 5497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:10.600669 5499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:10.600886 5491 server_base.cc:1034] running on GCE node
I20260620 19:08:10.601239 5491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:10.601502 5491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:10.602661 5491 hybrid_clock.cc:648] HybridClock initialized: now 1781982490602644 us; error 30 us; skew 500 ppm
I20260620 19:08:10.604035 5491 webserver.cc:460] Webserver started at http://127.0.203.1:46101/ using document root <none> and password file <none>
I20260620 19:08:10.604297 5491 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:10.604365 5491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:10.604519 5491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:10.605587 5491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "27b119032c4346f4afb3cb83a69e780f"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-wd30"
server_key: "9a3a52e4288ac8b87067bfbf993a9758"
server_key_iv: "22a0fad265b847874649562303ecbe9a"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.605952 5491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "27b119032c4346f4afb3cb83a69e780f"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-wd30"
server_key: "9a3a52e4288ac8b87067bfbf993a9758"
server_key_iv: "22a0fad265b847874649562303ecbe9a"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.607363 5491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:08:10.608209 5505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.608388 5491 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:10.608510 5491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "27b119032c4346f4afb3cb83a69e780f"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-wd30"
server_key: "9a3a52e4288ac8b87067bfbf993a9758"
server_key_iv: "22a0fad265b847874649562303ecbe9a"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.608588 5491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:10.611907 5491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:10.612196 5491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:10.612342 5491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:10.612565 5491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:10.612929 5491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:10.612985 5491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.613025 5491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:10.613052 5491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.618465 5491 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:43549
I20260620 19:08:10.618556 5618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:43549 every 8 connection(s)
I20260620 19:08:10.618884 5491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:08:10.621922 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.619633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:55583 (local address 127.0.203.62:35101)
0620 19:08:10.619674 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.619678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.619903 (+ 225us) server_negotiation.cc:407] Connection header received
0620 19:08:10.620417 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.620418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.620425 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.620487 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:10.621670 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:10.621671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:10.621672 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:10.621677 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:10.621692 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:10.621697 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:10.621750 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.621837 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.621853 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:08:10.622164 5621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.619615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:35101 (local address 127.0.203.1:55583)
0620 19:08:10.619782 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.619816 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.620303 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.620559 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.620562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.620572 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:10.621519 (+ 947us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:08:10.621520 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:10.621547 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:08:10.621575 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:10.621746 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:10.621751 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:10.621755 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.621849 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.621909 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260620 19:08:10.622627 5619 heartbeater.cc:347] Connected to a master server at 127.0.203.62:35101
I20260620 19:08:10.622704 5619 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:10.622853 5619 heartbeater.cc:511] Master 127.0.203.62:35101 requested a full tablet report, sending...
I20260620 19:08:10.623061 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 5491
I20260620 19:08:10.623186 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:08:10.623358 812 external_mini_cluster.cc:1428] Setting key b01078ce02a0e2925a4d9595b310bd72
I20260620 19:08:10.623524 5431 ts_manager.cc:194] Registered new tserver with Master: 27b119032c4346f4afb3cb83a69e780f (127.0.203.1:43549)
I20260620 19:08:10.624356 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:35101
--builtin_ntp_servers=127.0.203.20:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 19:08:10.705688 5622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:10.705890 5622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:10.705912 5622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:10.707093 5622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:10.707151 5622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:10.707178 5622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:08:10.708562 5622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:35101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:10.708810 5622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:10.709102 5622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:10.714126 5627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:10.714128 5628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:10.714236 5630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:10.714325 5622 server_base.cc:1034] running on GCE node
I20260620 19:08:10.714517 5622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:10.714777 5622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:10.715936 5622 hybrid_clock.cc:648] HybridClock initialized: now 1781982490715933 us; error 36 us; skew 500 ppm
I20260620 19:08:10.717104 5622 webserver.cc:460] Webserver started at http://127.0.203.2:45503/ using document root <none> and password file <none>
I20260620 19:08:10.717370 5622 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:10.717438 5622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:10.717612 5622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:10.718688 5622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "14c0cace0ebb4cb19e7e9318cb8a4cfe"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-wd30"
server_key: "37e16e2c247d56859301277fd5a750be"
server_key_iv: "bd179640ec4906ecdecfa00e973b3dfb"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.719056 5622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "14c0cace0ebb4cb19e7e9318cb8a4cfe"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-wd30"
server_key: "37e16e2c247d56859301277fd5a750be"
server_key_iv: "bd179640ec4906ecdecfa00e973b3dfb"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.720367 5622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:08:10.721129 5636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.721314 5622 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:08:10.721405 5622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "14c0cace0ebb4cb19e7e9318cb8a4cfe"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-wd30"
server_key: "37e16e2c247d56859301277fd5a750be"
server_key_iv: "bd179640ec4906ecdecfa00e973b3dfb"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.721494 5622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:10.724848 5622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:10.725129 5622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:10.725275 5622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:10.725492 5622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:10.725795 5622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:10.725848 5622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.725888 5622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:10.725914 5622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.731838 5622 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:45613
I20260620 19:08:10.731873 5749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:45613 every 8 connection(s)
I20260620 19:08:10.732197 5622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:08:10.735090 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.732968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:52705 (local address 127.0.203.62:35101)
0620 19:08:10.733054 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.733056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.733240 (+ 184us) server_negotiation.cc:407] Connection header received
0620 19:08:10.733706 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.733707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.733712 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.733762 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:10.734914 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:10.734915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:10.734915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:10.734917 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:10.734928 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:10.734931 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:10.734971 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.735026 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.735037 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:08:10.735345 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.732932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:35101 (local address 127.0.203.2:52705)
0620 19:08:10.733125 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.733158 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.733623 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.733813 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.733816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.733824 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:10.734784 (+ 960us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:08:10.734785 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:10.734812 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:08:10.734840 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:10.734971 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:10.734973 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:10.734975 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.735031 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.735078 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260620 19:08:10.735862 5750 heartbeater.cc:347] Connected to a master server at 127.0.203.62:35101
I20260620 19:08:10.735965 5750 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:10.736114 5750 heartbeater.cc:511] Master 127.0.203.62:35101 requested a full tablet report, sending...
I20260620 19:08:10.736479 5430 ts_manager.cc:194] Registered new tserver with Master: 14c0cace0ebb4cb19e7e9318cb8a4cfe (127.0.203.2:45613)
I20260620 19:08:10.739050 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 5622
I20260620 19:08:10.739193 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:08:10.739375 812 external_mini_cluster.cc:1428] Setting key 1dcb44060e577cafb92b0d55ff8d7a94
I20260620 19:08:10.740199 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:35101
--builtin_ntp_servers=127.0.203.20:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 19:08:10.820334 5753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:10.820546 5753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:10.820569 5753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:10.821746 5753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:10.821789 5753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:10.821813 5753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:08:10.823194 5753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:35101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:10.823446 5753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:10.823758 5753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:10.829190 5759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:10.829180 5758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:10.829178 5761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:10.829407 5753 server_base.cc:1034] running on GCE node
I20260620 19:08:10.829634 5753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:10.829926 5753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:10.831090 5753 hybrid_clock.cc:648] HybridClock initialized: now 1781982490831068 us; error 51 us; skew 500 ppm
I20260620 19:08:10.832412 5753 webserver.cc:460] Webserver started at http://127.0.203.3:34837/ using document root <none> and password file <none>
I20260620 19:08:10.832623 5753 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:10.832666 5753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:10.832756 5753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:10.833797 5753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "5f3d36c360574ce0afcf0dc4d4f52f03"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-wd30"
server_key: "6ac77d2ec01694bdb239420c56195c03"
server_key_iv: "d6ea3ae9e6954f590b465d4c93b16840"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.834115 5753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "5f3d36c360574ce0afcf0dc4d4f52f03"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-wd30"
server_key: "6ac77d2ec01694bdb239420c56195c03"
server_key_iv: "d6ea3ae9e6954f590b465d4c93b16840"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.835462 5753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:08:10.836170 5767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.836362 5753 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:08:10.836436 5753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "5f3d36c360574ce0afcf0dc4d4f52f03"
format_stamp: "Formatted at 2026-06-20 19:08:10 on dist-test-slave-wd30"
server_key: "6ac77d2ec01694bdb239420c56195c03"
server_key_iv: "d6ea3ae9e6954f590b465d4c93b16840"
server_key_version: "encryptionkey@0"
I20260620 19:08:10.836534 5753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:10.839959 5753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:10.840231 5753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:10.840370 5753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:10.840572 5753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:10.840868 5753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:10.840901 5753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.840922 5753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:10.840968 5753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:10.846663 5753 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:43859
I20260620 19:08:10.846707 5880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:43859 every 8 connection(s)
I20260620 19:08:10.847038 5753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:08:10.850086 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.847833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:54707 (local address 127.0.203.62:35101)
0620 19:08:10.847894 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.847896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.848167 (+ 271us) server_negotiation.cc:407] Connection header received
0620 19:08:10.848625 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.848626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.848631 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.848690 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:10.849882 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:10.849883 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:10.849884 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:10.849887 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:10.849900 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:10.849904 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:10.849953 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.850019 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.850031 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:08:10.850267 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.847831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:35101 (local address 127.0.203.3:54707)
0620 19:08:10.848027 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.848069 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.848540 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.848745 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.848748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.848757 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:10.849728 (+ 971us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:08:10.849728 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:10.849756 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:08:10.849785 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:10.849951 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:10.849954 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:10.849956 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.850008 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.850064 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260620 19:08:10.850865 5881 heartbeater.cc:347] Connected to a master server at 127.0.203.62:35101
I20260620 19:08:10.850957 5881 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:10.851176 5881 heartbeater.cc:511] Master 127.0.203.62:35101 requested a full tablet report, sending...
I20260620 19:08:10.851559 5430 ts_manager.cc:194] Registered new tserver with Master: 5f3d36c360574ce0afcf0dc4d4f52f03 (127.0.203.3:43859)
I20260620 19:08:10.854763 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 5753
I20260620 19:08:10.854848 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:08:10.855054 812 external_mini_cluster.cc:1428] Setting key 40ed5704ea3cbe97981368267c337629
I20260620 19:08:10.855906 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:08:10.858372 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.857291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59422 (local address 127.0.203.62:35101)
0620 19:08:10.857359 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.857360 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.857480 (+ 120us) server_negotiation.cc:407] Connection header received
0620 19:08:10.857525 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.857526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.857531 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.857590 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:10.858210 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:10.858211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:10.858211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:10.858214 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:10.858227 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:10.858230 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:10.858277 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.858310 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.858322 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:08:10.860153 5430 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:08:10.861791 5430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:08:10.865583 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.864067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:43549 (local address 127.0.0.1:53262)
0620 19:08:10.864449 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.864458 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.864489 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.864837 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.864838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.864841 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:10.865186 (+ 345us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:08:10.865186 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:10.865191 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:08:10.865200 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:10.865447 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:10.865449 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:10.865449 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.865482 (+ 33us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.865497 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":95,"threads_started":1}
I20260620 19:08:10.865604 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.864150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.0.203.2:45613)
0620 19:08:10.864329 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.864336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.864365 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:08:10.864392 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.864394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.864407 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.864489 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:10.865186 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:10.865199 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:10.865200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:10.865204 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:10.865241 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:10.865274 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:10.865335 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.865478 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.865497 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
I20260620 19:08:10.865823 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.864045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:45613 (local address 127.0.0.1:57374)
0620 19:08:10.864229 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.864272 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.864328 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.864504 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.864510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.864516 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:10.865033 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:08:10.865034 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:10.865050 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:08:10.865096 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:10.865362 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:10.865369 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:10.865372 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.865720 (+ 348us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.865734 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":64,"threads_started":1}
I20260620 19:08:10.865965 5890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.864066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:43859 (local address 127.0.0.1:48592)
0620 19:08:10.864307 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.864318 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.864368 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.865069 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.865070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.865073 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:10.865465 (+ 392us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:08:10.865466 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:10.865472 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:08:10.865487 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:10.865829 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:10.865830 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:10.865831 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.865884 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.865896 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":101,"threads_started":1}
I20260620 19:08:10.866139 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.864493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53262 (local address 127.0.203.1:43549)
0620 19:08:10.864685 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.864690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.864701 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:08:10.864728 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.864730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.864739 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.864817 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:10.865296 (+ 479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:10.865303 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:10.865303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:10.865308 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:10.865342 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:10.865369 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:10.865996 (+ 627us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.866022 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.866039 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
I20260620 19:08:10.866411 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.864295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.0.203.3:43859)
0620 19:08:10.864913 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.864916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.864925 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:08:10.864950 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.864952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.864960 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.865269 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:10.865688 (+ 419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:10.865692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:10.865692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:10.865696 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:10.865726 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:10.865739 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:10.865826 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.865964 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.865981 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":81,"threads_started":1}
I20260620 19:08:10.866644 5684 tablet_service.cc:1467] Processing CreateTablet for tablet eeccd3b00a7541cda039823ebc842926 (DEFAULT_TABLE table=test-table [id=026ef039cb0b41dca6e35f34996bfb82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:10.866765 5553 tablet_service.cc:1467] Processing CreateTablet for tablet eeccd3b00a7541cda039823ebc842926 (DEFAULT_TABLE table=test-table [id=026ef039cb0b41dca6e35f34996bfb82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:10.867043 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeccd3b00a7541cda039823ebc842926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:10.867022 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeccd3b00a7541cda039823ebc842926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:10.867008 5815 tablet_service.cc:1467] Processing CreateTablet for tablet eeccd3b00a7541cda039823ebc842926 (DEFAULT_TABLE table=test-table [id=026ef039cb0b41dca6e35f34996bfb82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:10.867331 5815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeccd3b00a7541cda039823ebc842926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:10.869666 5895 tablet_bootstrap.cc:492] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f: Bootstrap starting.
I20260620 19:08:10.870344 5895 tablet_bootstrap.cc:654] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:10.870594 5896 tablet_bootstrap.cc:492] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe: Bootstrap starting.
I20260620 19:08:10.870754 5895 log.cc:824] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:10.871060 5897 tablet_bootstrap.cc:492] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03: Bootstrap starting.
I20260620 19:08:10.871272 5896 tablet_bootstrap.cc:654] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:10.871559 5895 tablet_bootstrap.cc:492] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f: No bootstrap required, opened a new log
I20260620 19:08:10.871603 5896 log.cc:824] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:10.871642 5895 ts_tablet_manager.cc:1397] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:08:10.871719 5897 tablet_bootstrap.cc:654] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:10.872025 5897 log.cc:824] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:10.872367 5896 tablet_bootstrap.cc:492] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe: No bootstrap required, opened a new log
I20260620 19:08:10.872447 5896 ts_tablet_manager.cc:1397] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:08:10.872673 5897 tablet_bootstrap.cc:492] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03: No bootstrap required, opened a new log
I20260620 19:08:10.872751 5897 ts_tablet_manager.cc:1397] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:08:10.874366 5895 raft_consensus.cc:348] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c0cace0ebb4cb19e7e9318cb8a4cfe" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45613 } } peers { permanent_uuid: "27b119032c4346f4afb3cb83a69e780f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43549 } } peers { permanent_uuid: "5f3d36c360574ce0afcf0dc4d4f52f03" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43859 } }
I20260620 19:08:10.874593 5895 raft_consensus.cc:374] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:10.874644 5895 raft_consensus.cc:729] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27b119032c4346f4afb3cb83a69e780f, State: Initialized, Role: FOLLOWER
I20260620 19:08:10.874631 5897 raft_consensus.cc:348] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c0cace0ebb4cb19e7e9318cb8a4cfe" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45613 } } peers { permanent_uuid: "27b119032c4346f4afb3cb83a69e780f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43549 } } peers { permanent_uuid: "5f3d36c360574ce0afcf0dc4d4f52f03" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43859 } }
I20260620 19:08:10.874790 5897 raft_consensus.cc:374] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:10.874774 5895 consensus_queue.cc:260] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c0cace0ebb4cb19e7e9318cb8a4cfe" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45613 } } peers { permanent_uuid: "27b119032c4346f4afb3cb83a69e780f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43549 } } peers { permanent_uuid: "5f3d36c360574ce0afcf0dc4d4f52f03" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43859 } }
I20260620 19:08:10.874830 5897 raft_consensus.cc:729] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f3d36c360574ce0afcf0dc4d4f52f03, State: Initialized, Role: FOLLOWER
I20260620 19:08:10.874970 5897 consensus_queue.cc:260] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c0cace0ebb4cb19e7e9318cb8a4cfe" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45613 } } peers { permanent_uuid: "27b119032c4346f4afb3cb83a69e780f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43549 } } peers { permanent_uuid: "5f3d36c360574ce0afcf0dc4d4f52f03" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43859 } }
I20260620 19:08:10.875032 5895 ts_tablet_manager.cc:1428] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 19:08:10.875175 5619 heartbeater.cc:503] Master 127.0.203.62:35101 was elected leader, sending a full tablet report...
I20260620 19:08:10.875056 5896 raft_consensus.cc:348] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c0cace0ebb4cb19e7e9318cb8a4cfe" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45613 } } peers { permanent_uuid: "27b119032c4346f4afb3cb83a69e780f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43549 } } peers { permanent_uuid: "5f3d36c360574ce0afcf0dc4d4f52f03" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43859 } }
I20260620 19:08:10.875217 5896 raft_consensus.cc:374] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:10.875243 5897 ts_tablet_manager.cc:1428] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 19:08:10.875255 5896 raft_consensus.cc:729] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c0cace0ebb4cb19e7e9318cb8a4cfe, State: Initialized, Role: FOLLOWER
I20260620 19:08:10.875357 5881 heartbeater.cc:503] Master 127.0.203.62:35101 was elected leader, sending a full tablet report...
I20260620 19:08:10.875553 5896 consensus_queue.cc:260] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c0cace0ebb4cb19e7e9318cb8a4cfe" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45613 } } peers { permanent_uuid: "27b119032c4346f4afb3cb83a69e780f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43549 } } peers { permanent_uuid: "5f3d36c360574ce0afcf0dc4d4f52f03" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43859 } }
I20260620 19:08:10.875790 5896 ts_tablet_manager.cc:1428] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 19:08:10.875890 5750 heartbeater.cc:503] Master 127.0.203.62:35101 was elected leader, sending a full tablet report...
I20260620 19:08:10.881026 5901 raft_consensus.cc:482] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:08:10.881137 5901 raft_consensus.cc:504] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c0cace0ebb4cb19e7e9318cb8a4cfe" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45613 } } peers { permanent_uuid: "27b119032c4346f4afb3cb83a69e780f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43549 } } peers { permanent_uuid: "5f3d36c360574ce0afcf0dc4d4f52f03" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43859 } }
I20260620 19:08:10.881474 5901 leader_election.cc:288] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14c0cace0ebb4cb19e7e9318cb8a4cfe (127.0.203.2:45613), 5f3d36c360574ce0afcf0dc4d4f52f03 (127.0.203.3:43859)
I20260620 19:08:10.883208 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.881800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:46483 (local address 127.0.203.3:43859)
0620 19:08:10.881843 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.881845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.881903 (+ 58us) server_negotiation.cc:407] Connection header received
0620 19:08:10.881990 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.881991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.881995 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.882048 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:10.882926 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:10.882927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:10.882927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:10.882931 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:10.882948 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:10.882953 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:10.883047 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.883137 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.883151 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:08:10.883224 5904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.881666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:43859 (local address 127.0.203.1:46483)
0620 19:08:10.881848 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.881857 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.881922 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.882065 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.882066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.882068 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:10.882765 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:08:10.882765 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:10.882772 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:08:10.882785 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:10.883047 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:10.883047 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:10.883049 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.883107 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.883137 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":19,"thread_start_us":99,"threads_started":1}
I20260620 19:08:10.883307 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.881894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:34527 (local address 127.0.203.2:45613)
0620 19:08:10.881939 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.881941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.881949 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:08:10.881972 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.881973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.881978 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.882038 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:10.883001 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:10.883002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:10.883003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:10.883008 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:10.883024 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:10.883029 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:10.883101 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.883233 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.883248 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:08:10.883306 5621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.881663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:45613 (local address 127.0.203.1:34527)
0620 19:08:10.881748 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.881770 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:10.881810 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:10.882097 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:10.882098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:10.882100 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:10.882875 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:08:10.882875 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:10.882881 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:08:10.882893 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:10.883100 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:10.883101 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:10.883101 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 19:08:10.883227 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 19:08:10.883242 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 19:08:10.883682 5835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeccd3b00a7541cda039823ebc842926" candidate_uuid: "27b119032c4346f4afb3cb83a69e780f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f3d36c360574ce0afcf0dc4d4f52f03" is_pre_election: true
I20260620 19:08:10.883740 5704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeccd3b00a7541cda039823ebc842926" candidate_uuid: "27b119032c4346f4afb3cb83a69e780f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c0cace0ebb4cb19e7e9318cb8a4cfe" is_pre_election: true
I20260620 19:08:10.883834 5835 raft_consensus.cc:2454] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27b119032c4346f4afb3cb83a69e780f in term 0.
I20260620 19:08:10.883841 5704 raft_consensus.cc:2454] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27b119032c4346f4afb3cb83a69e780f in term 0.
I20260620 19:08:10.884058 5508 leader_election.cc:302] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27b119032c4346f4afb3cb83a69e780f, 5f3d36c360574ce0afcf0dc4d4f52f03; no voters:
I20260620 19:08:10.884222 5901 raft_consensus.cc:2781] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:08:10.884271 5901 raft_consensus.cc:482] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:08:10.884286 5901 raft_consensus.cc:3037] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:10.884959 5901 raft_consensus.cc:504] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c0cace0ebb4cb19e7e9318cb8a4cfe" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45613 } } peers { permanent_uuid: "27b119032c4346f4afb3cb83a69e780f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43549 } } peers { permanent_uuid: "5f3d36c360574ce0afcf0dc4d4f52f03" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43859 } }
I20260620 19:08:10.885106 5901 leader_election.cc:288] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [CANDIDATE]: Term 1 election: Requested vote from peers 14c0cace0ebb4cb19e7e9318cb8a4cfe (127.0.203.2:45613), 5f3d36c360574ce0afcf0dc4d4f52f03 (127.0.203.3:43859)
I20260620 19:08:10.885241 5835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeccd3b00a7541cda039823ebc842926" candidate_uuid: "27b119032c4346f4afb3cb83a69e780f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f3d36c360574ce0afcf0dc4d4f52f03"
I20260620 19:08:10.885289 5704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeccd3b00a7541cda039823ebc842926" candidate_uuid: "27b119032c4346f4afb3cb83a69e780f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c0cace0ebb4cb19e7e9318cb8a4cfe"
I20260620 19:08:10.885341 5835 raft_consensus.cc:3037] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:10.885346 5704 raft_consensus.cc:3037] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:10.886042 5835 raft_consensus.cc:2454] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27b119032c4346f4afb3cb83a69e780f in term 1.
I20260620 19:08:10.886044 5704 raft_consensus.cc:2454] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27b119032c4346f4afb3cb83a69e780f in term 1.
I20260620 19:08:10.886233 5508 leader_election.cc:302] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27b119032c4346f4afb3cb83a69e780f, 5f3d36c360574ce0afcf0dc4d4f52f03; no voters:
I20260620 19:08:10.886317 5901 raft_consensus.cc:2781] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:10.886507 5901 raft_consensus.cc:686] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [term 1 LEADER]: Becoming Leader. State: Replica: 27b119032c4346f4afb3cb83a69e780f, State: Running, Role: LEADER
I20260620 19:08:10.886626 5901 consensus_queue.cc:237] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c0cace0ebb4cb19e7e9318cb8a4cfe" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45613 } } peers { permanent_uuid: "27b119032c4346f4afb3cb83a69e780f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43549 } } peers { permanent_uuid: "5f3d36c360574ce0afcf0dc4d4f52f03" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43859 } }
I20260620 19:08:10.887444 5430 catalog_manager.cc:5495] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f reported cstate change: term changed from 0 to 1, leader changed from <none> to 27b119032c4346f4afb3cb83a69e780f (127.0.203.1). New cstate: current_term: 1 leader_uuid: "27b119032c4346f4afb3cb83a69e780f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c0cace0ebb4cb19e7e9318cb8a4cfe" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27b119032c4346f4afb3cb83a69e780f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f3d36c360574ce0afcf0dc4d4f52f03" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43859 } health_report { overall_health: UNKNOWN } } }
I20260620 19:08:10.894157 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.893123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53270 (local address 127.0.203.1:43549)
0620 19:08:10.893188 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:10.893191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:10.893265 (+ 74us) server_negotiation.cc:407] Connection header received
0620 19:08:10.893368 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:10.893369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:10.893383 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:10.893430 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:10.893934 (+ 504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:10.893935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:10.893935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:10.893940 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:10.893954 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:10.893958 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:10.894006 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:10.894069 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 19:08:10.894083 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:08:10.896036 5835 raft_consensus.cc:1261] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03 [term 1 FOLLOWER]: Refusing update from remote peer 27b119032c4346f4afb3cb83a69e780f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:10.896085 5704 raft_consensus.cc:1261] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe [term 1 FOLLOWER]: Refusing update from remote peer 27b119032c4346f4afb3cb83a69e780f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:10.896371 5901 consensus_queue.cc:1035] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f3d36c360574ce0afcf0dc4d4f52f03" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:10.896528 5905 consensus_queue.cc:1035] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c0cace0ebb4cb19e7e9318cb8a4cfe" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:10.899104 5908 mvcc.cc:205] Tried to move back new op lower bound from 7299000282707021824 to 7299000282672279552. Current Snapshot: MvccSnapshot[applied={T|T < 7299000282707021824 or (T in {7299000282707021824})}]
I20260620 19:08:10.900329 5911 mvcc.cc:205] Tried to move back new op lower bound from 7299000282707021824 to 7299000282672279552. Current Snapshot: MvccSnapshot[applied={T|T < 7299000282707021824}]
I20260620 19:08:10.901098 5430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:10.901196 5430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:10.901957 5430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06bd1d99e67a481db3c58620a5022d9f: Sending DeleteTablet for 3 replicas of tablet eeccd3b00a7541cda039823ebc842926
I20260620 19:08:10.902254 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet eeccd3b00a7541cda039823ebc842926 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:10 UTC) from {username='slave'} at 127.0.0.1:57374
I20260620 19:08:10.902289 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet eeccd3b00a7541cda039823ebc842926 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:10 UTC) from {username='slave'} at 127.0.0.1:53262
I20260620 19:08:10.902367 5815 tablet_service.cc:1514] Processing DeleteTablet for tablet eeccd3b00a7541cda039823ebc842926 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:10 UTC) from {username='slave'} at 127.0.0.1:48592
I20260620 19:08:10.902514 5919 tablet_replica.cc:330] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f: stopping tablet replica
I20260620 19:08:10.902514 5918 tablet_replica.cc:330] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe: stopping tablet replica
I20260620 19:08:10.902609 5918 raft_consensus.cc:2229] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:10.902618 5919 raft_consensus.cc:2229] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:08:10.902701 5919 raft_consensus.cc:2258] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:10.902711 5918 raft_consensus.cc:2258] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:10.903054 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 5491
I20260620 19:08:10.903177 5918 ts_tablet_manager.cc:1905] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:10.903378 5919 ts_tablet_manager.cc:1905] T eeccd3b00a7541cda039823ebc842926 P 27b119032c4346f4afb3cb83a69e780f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:10.903676 5920 tablet_replica.cc:330] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03: stopping tablet replica
I20260620 19:08:10.903801 5920 raft_consensus.cc:2229] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:10.903903 5920 raft_consensus.cc:2258] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:10.904268 5920 ts_tablet_manager.cc:1905] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:10.905275 5918 ts_tablet_manager.cc:1918] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:10.905354 5918 log.cc:1196] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/eeccd3b00a7541cda039823ebc842926
I20260620 19:08:10.905630 5918 ts_tablet_manager.cc:1939] T eeccd3b00a7541cda039823ebc842926 P 14c0cace0ebb4cb19e7e9318cb8a4cfe: Deleting consensus metadata
I20260620 19:08:10.905642 5920 ts_tablet_manager.cc:1918] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:10.905715 5920 log.cc:1196] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/eeccd3b00a7541cda039823ebc842926
I20260620 19:08:10.905949 5920 ts_tablet_manager.cc:1939] T eeccd3b00a7541cda039823ebc842926 P 5f3d36c360574ce0afcf0dc4d4f52f03: Deleting consensus metadata
I20260620 19:08:10.906064 5418 catalog_manager.cc:4841] TS 14c0cace0ebb4cb19e7e9318cb8a4cfe (127.0.203.2:45613): tablet eeccd3b00a7541cda039823ebc842926 (table test-table [id=026ef039cb0b41dca6e35f34996bfb82]) successfully deleted
I20260620 19:08:10.906375 5417 catalog_manager.cc:4841] TS 5f3d36c360574ce0afcf0dc4d4f52f03 (127.0.203.3:43859): tablet eeccd3b00a7541cda039823ebc842926 (table test-table [id=026ef039cb0b41dca6e35f34996bfb82]) successfully deleted
W20260620 19:08:10.910240 5416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.1:43549 (error 108)
I20260620 19:08:10.910421 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 5622
I20260620 19:08:10.910885 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:10.910761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:43549 (local address 127.0.0.1:53282)
0620 19:08:10.910808 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:10.910843 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.1:43549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 19:08:10.910996 5416 catalog_manager.cc:4584] TS 27b119032c4346f4afb3cb83a69e780f (127.0.203.1:43549): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eeccd3b00a7541cda039823ebc842926: Network error: Client connection negotiation failed: client connection to 127.0.203.1:43549: connect: Connection refused (error 111)
I20260620 19:08:10.914925 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 5753
I20260620 19:08:10.919737 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 5400
2026-06-20T19:08:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (572 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 19:08:10.939569 812 mini_oidc.cc:152] Starting JWKS server
I20260620 19:08:10.940588 812 webserver.cc:460] Webserver started at https://127.0.0.1:37095/ using document root <none> and password file <none>
I20260620 19:08:10.940675 812 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 19:08:10.940802 812 webserver.cc:460] Webserver started at http://127.0.0.1:37975/ using document root <none> and password file <none>
2026-06-20T19:08:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:08:10Z Disabled control of system clock
I20260620 19:08:10.948676 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:42721
--webserver_interface=127.0.203.62
--webserver_port=0
--builtin_ntp_servers=127.0.203.20:34417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:42721
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37095/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982468536772-812-0/testchainca.cert with env {}
W20260620 19:08:11.027980 5927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:11.028211 5927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:11.028234 5927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:11.029353 5927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:11.029385 5927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:11.029398 5927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:11.029410 5927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:11.029422 5927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:08:11.029434 5927 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:08:11.029445 5927 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37095/jwks.json
I20260620 19:08:11.030874 5927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:34417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:42721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:42721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37095/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982468536772-812-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:11.031059 5927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:11.031284 5927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:11.035934 5933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:11.035930 5935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:11.036145 5927 server_base.cc:1034] running on GCE node
W20260620 19:08:11.035959 5932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:11.036450 5927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:11.036757 5927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:11.037902 5927 hybrid_clock.cc:648] HybridClock initialized: now 1781982491037882 us; error 41 us; skew 500 ppm
I20260620 19:08:11.039072 5927 webserver.cc:460] Webserver started at http://127.0.203.62:40493/ using document root <none> and password file <none>
I20260620 19:08:11.039341 5927 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:11.039410 5927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:11.039556 5927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:11.040607 5927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "93d9d13322fb4128a9cf66de3116d7b0"
format_stamp: "Formatted at 2026-06-20 19:08:11 on dist-test-slave-wd30"
server_key: "009b781b581773476bc3c5b9bbd11a1a"
server_key_iv: "5e8b3b0b40c3203e5989c744076fbaf7"
server_key_version: "encryptionkey@0"
I20260620 19:08:11.041004 5927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "93d9d13322fb4128a9cf66de3116d7b0"
format_stamp: "Formatted at 2026-06-20 19:08:11 on dist-test-slave-wd30"
server_key: "009b781b581773476bc3c5b9bbd11a1a"
server_key_iv: "5e8b3b0b40c3203e5989c744076fbaf7"
server_key_version: "encryptionkey@0"
I20260620 19:08:11.042317 5927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:08:11.043027 5941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:11.043228 5927 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:11.043334 5927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "93d9d13322fb4128a9cf66de3116d7b0"
format_stamp: "Formatted at 2026-06-20 19:08:11 on dist-test-slave-wd30"
server_key: "009b781b581773476bc3c5b9bbd11a1a"
server_key_iv: "5e8b3b0b40c3203e5989c744076fbaf7"
server_key_version: "encryptionkey@0"
I20260620 19:08:11.043407 5927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:11.059981 5927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:11.060330 5927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:11.060489 5927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:11.064471 5927 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:42721
I20260620 19:08:11.064508 5993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:42721 every 8 connection(s)
I20260620 19:08:11.064885 5927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:11.065820 5994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:11.068156 5994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0: Bootstrap starting.
I20260620 19:08:11.068801 5994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:11.069104 5994 log.cc:824] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:11.069792 5994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0: No bootstrap required, opened a new log
I20260620 19:08:11.071450 5994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d9d13322fb4128a9cf66de3116d7b0" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42721 } }
I20260620 19:08:11.071601 5994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:11.071625 5994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93d9d13322fb4128a9cf66de3116d7b0, State: Initialized, Role: FOLLOWER
I20260620 19:08:11.071800 5994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d9d13322fb4128a9cf66de3116d7b0" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42721 } }
I20260620 19:08:11.071861 5994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:11.071882 5994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:11.071952 5994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:11.072466 5994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d9d13322fb4128a9cf66de3116d7b0" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42721 } }
I20260620 19:08:11.072577 5994 leader_election.cc:302] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93d9d13322fb4128a9cf66de3116d7b0; no voters:
I20260620 19:08:11.072804 5994 leader_election.cc:288] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:08:11.072837 5997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:11.072887 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 5927
I20260620 19:08:11.072953 5997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0 [term 1 LEADER]: Becoming Leader. State: Replica: 93d9d13322fb4128a9cf66de3116d7b0, State: Running, Role: LEADER
I20260620 19:08:11.072991 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:08:11.073067 5994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:11.073101 5997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d9d13322fb4128a9cf66de3116d7b0" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42721 } }
I20260620 19:08:11.073218 812 external_mini_cluster.cc:1428] Setting key 2ab15231723d596d41e9ef9391fb3030
I20260620 19:08:11.073406 5997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93d9d13322fb4128a9cf66de3116d7b0. Latest consensus state: current_term: 1 leader_uuid: "93d9d13322fb4128a9cf66de3116d7b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d9d13322fb4128a9cf66de3116d7b0" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42721 } } }
I20260620 19:08:11.073483 5997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:11.073873 5998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93d9d13322fb4128a9cf66de3116d7b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d9d13322fb4128a9cf66de3116d7b0" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42721 } } }
I20260620 19:08:11.074004 5998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:11.074399 6003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:08:11.074950 6003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:08:11.076586 6003 catalog_manager.cc:1347] Generated new cluster ID: 3ab3928a46084e8b8544aec2c23eb171
I20260620 19:08:11.076638 6003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:08:11.079936 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:11.073806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54620 (local address 127.0.203.62:42721)
0620 19:08:11.074042 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:11.074060 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:11.074257 (+ 197us) server_negotiation.cc:407] Connection header received
0620 19:08:11.074488 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:11.074496 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:11.074576 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:11.074880 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:11.075089 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.076596 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.077726 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.078324 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.078442 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:11.079228 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:11.079232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:11.079232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:11.079430 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:11.079469 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:11.079482 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:11.079543 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:11.079656 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 19:08:11.079730 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":104,"threads_started":1}
I20260620 19:08:11.080863 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:08:11.099922 6003 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:08:11.100674 6003 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:11.107590 6003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93d9d13322fb4128a9cf66de3116d7b0: Generated new TSK 0
I20260620 19:08:11.107797 6003 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 19:08:11.131778 6024 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 19:08:11.131886 6024 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 19:08:11.132174 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:11.116895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54622 (local address 127.0.203.62:42721)
0620 19:08:11.116937 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:11.116939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:11.117056 (+ 117us) server_negotiation.cc:407] Connection header received
0620 19:08:11.117164 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:11.117165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:11.117176 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:11.117233 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:08:11.117384 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.117877 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.118991 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.119222 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.119273 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:11.119494 (+ 221us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:08:11.131981 (+ 12487us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 19:08:11.132121 (+ 140us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54622: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 19:08:11.132239 6008 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54622: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 19:08:11.132570 6023 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:11.116788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42721 (local address 127.0.0.1:54622)
0620 19:08:11.116977 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:11.116990 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:11.117069 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:11.117240 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:11.117241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:11.117243 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:08:11.117318 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:11.117321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.117962 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.117963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:11.118890 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:11.118893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.119293 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.119294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:11.119363 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:11.119385 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:08:11.132104 (+ 12719us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 19:08:11.132242 (+ 138us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.203.62:42721: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.203.62:42721: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 19:08:11.133035 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 5927
2026-06-20T19:08:11Z chronyd exiting
I20260620 19:08:11.168917 812 test_util.cc:175] -----------------------------------------------
I20260620 19:08:11.169018 812 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982468536772-812-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (230 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 19:08:11.169798 812 mini_oidc.cc:152] Starting JWKS server
I20260620 19:08:11.170463 812 webserver.cc:460] Webserver started at https://127.0.0.1:37367/ using document root <none> and password file <none>
I20260620 19:08:11.170531 812 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 19:08:11.170725 812 webserver.cc:460] Webserver started at http://127.0.0.1:41323/ using document root <none> and password file <none>
2026-06-20T19:08:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:08:11Z Disabled control of system clock
I20260620 19:08:11.178066 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:42507
--webserver_interface=127.0.203.62
--webserver_port=0
--builtin_ntp_servers=127.0.203.20:40815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:42507
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37367/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982468536772-812-0/testchainca.cert with env {}
W20260620 19:08:11.257005 6030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:11.257206 6030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:11.257228 6030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:11.258306 6030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:11.258339 6030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:11.258352 6030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:11.258365 6030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:11.258378 6030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:08:11.258389 6030 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:08:11.258401 6030 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37367/jwks.json
I20260620 19:08:11.259855 6030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:42507
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:42507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37367/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982468536772-812-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:11.260043 6030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:11.260206 6030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:11.264850 6035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:11.264928 6036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:11.264940 6038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:11.265017 6030 server_base.cc:1034] running on GCE node
I20260620 19:08:11.265381 6030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:11.265697 6030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:11.266873 6030 hybrid_clock.cc:648] HybridClock initialized: now 1781982491266862 us; error 47 us; skew 500 ppm
I20260620 19:08:11.268142 6030 webserver.cc:460] Webserver started at http://127.0.203.62:42655/ using document root <none> and password file <none>
I20260620 19:08:11.268385 6030 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:11.268452 6030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:11.268602 6030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:11.269654 6030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "380e83c8225a428f979efb1a7f16c70e"
format_stamp: "Formatted at 2026-06-20 19:08:11 on dist-test-slave-wd30"
server_key: "89bbcf301981c8847f7630047e01c9ad"
server_key_iv: "906344338edcfa658bb0fd231af42956"
server_key_version: "encryptionkey@0"
I20260620 19:08:11.270021 6030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "380e83c8225a428f979efb1a7f16c70e"
format_stamp: "Formatted at 2026-06-20 19:08:11 on dist-test-slave-wd30"
server_key: "89bbcf301981c8847f7630047e01c9ad"
server_key_iv: "906344338edcfa658bb0fd231af42956"
server_key_version: "encryptionkey@0"
I20260620 19:08:11.271427 6030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:08:11.272167 6044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:11.272331 6030 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:11.272434 6030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "380e83c8225a428f979efb1a7f16c70e"
format_stamp: "Formatted at 2026-06-20 19:08:11 on dist-test-slave-wd30"
server_key: "89bbcf301981c8847f7630047e01c9ad"
server_key_iv: "906344338edcfa658bb0fd231af42956"
server_key_version: "encryptionkey@0"
I20260620 19:08:11.272508 6030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:11.279539 6030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:11.279814 6030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:11.279955 6030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:11.283859 6030 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:42507
I20260620 19:08:11.283898 6096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:42507 every 8 connection(s)
I20260620 19:08:11.284245 6030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:11.285198 6097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:11.287441 6097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e: Bootstrap starting.
I20260620 19:08:11.287954 6097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:11.288182 6097 log.cc:824] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:11.288828 6097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e: No bootstrap required, opened a new log
I20260620 19:08:11.290309 6097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380e83c8225a428f979efb1a7f16c70e" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42507 } }
I20260620 19:08:11.290442 6097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:11.290464 6097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 380e83c8225a428f979efb1a7f16c70e, State: Initialized, Role: FOLLOWER
I20260620 19:08:11.290597 6097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380e83c8225a428f979efb1a7f16c70e" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42507 } }
I20260620 19:08:11.290683 6097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:11.290709 6097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:11.290736 6097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:11.291271 6097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380e83c8225a428f979efb1a7f16c70e" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42507 } }
I20260620 19:08:11.291355 6097 leader_election.cc:302] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 380e83c8225a428f979efb1a7f16c70e; no voters:
I20260620 19:08:11.291477 6097 leader_election.cc:288] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:08:11.291560 6100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:11.291729 6100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e [term 1 LEADER]: Becoming Leader. State: Replica: 380e83c8225a428f979efb1a7f16c70e, State: Running, Role: LEADER
I20260620 19:08:11.291752 6097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:11.291829 6100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380e83c8225a428f979efb1a7f16c70e" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42507 } }
I20260620 19:08:11.292052 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 6030
I20260620 19:08:11.292160 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:08:11.292222 6101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "380e83c8225a428f979efb1a7f16c70e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380e83c8225a428f979efb1a7f16c70e" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42507 } } }
I20260620 19:08:11.292361 812 external_mini_cluster.cc:1428] Setting key a391e51a33abe2ae555c1a2e542be387
I20260620 19:08:11.292312 6102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 380e83c8225a428f979efb1a7f16c70e. Latest consensus state: current_term: 1 leader_uuid: "380e83c8225a428f979efb1a7f16c70e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380e83c8225a428f979efb1a7f16c70e" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42507 } } }
I20260620 19:08:11.292338 6101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:11.292402 6102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:11.292762 6104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:08:11.293316 6104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:08:11.294903 6104 catalog_manager.cc:1347] Generated new cluster ID: 49557f35c98f46858bbbc8edc7484f73
I20260620 19:08:11.294952 6104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:08:11.298602 6107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:11.292763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.0.203.62:42507)
0620 19:08:11.294083 (+ 1320us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:11.294093 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:11.294129 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:08:11.294261 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:11.294269 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:11.294457 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:11.294607 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:11.294868 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.295921 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.296714 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.297118 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.297271 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:11.297965 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:11.297969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:11.297969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:11.298139 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:11.298177 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:11.298187 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:11.298258 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:11.298329 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 19:08:11.298401 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1244,"thread_start_us":69,"threads_started":1}
I20260620 19:08:11.299463 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:08:11.322077 6104 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:08:11.322685 6104 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:11.328457 6104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 380e83c8225a428f979efb1a7f16c70e: Generated new TSK 0
I20260620 19:08:11.328640 6104 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 19:08:11.357571 6127 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 19:08:11.357666 6127 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 19:08:11.357976 6107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:11.335968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48834 (local address 127.0.203.62:42507)
0620 19:08:11.336018 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:11.336021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:11.336107 (+ 86us) server_negotiation.cc:407] Connection header received
0620 19:08:11.336184 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:11.336185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:11.336195 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:11.336254 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:08:11.336412 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.336907 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.338034 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.338236 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.338285 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:11.338477 (+ 192us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:08:11.357727 (+ 19250us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 19:08:11.357886 (+ 159us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48834: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":28}
W20260620 19:08:11.358026 6107 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48834: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 19:08:11.358106 6126 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:11.335884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42507 (local address 127.0.0.1:48834)
0620 19:08:11.336048 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:11.336060 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:11.336125 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:11.336280 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:11.336281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:11.336283 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:08:11.336344 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:11.336346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.336988 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.336988 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:11.337950 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:11.337953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.338319 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.338320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:11.338381 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:11.338387 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:08:11.357879 (+ 19492us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 19:08:11.358011 (+ 132us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.203.62:42507: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260620 19:08:11.358506 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 6030
2026-06-20T19:08:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (226 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 19:08:11.396548 812 mini_oidc.cc:152] Starting JWKS server
I20260620 19:08:11.397225 812 webserver.cc:460] Webserver started at https://127.0.0.1:41863/ using document root <none> and password file <none>
I20260620 19:08:11.397266 812 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 19:08:11.397375 812 webserver.cc:460] Webserver started at http://127.0.0.1:35845/ using document root <none> and password file <none>
2026-06-20T19:08:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:08:11Z Disabled control of system clock
I20260620 19:08:11.405216 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:33333
--webserver_interface=127.0.203.62
--webserver_port=0
--builtin_ntp_servers=127.0.203.20:40167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:33333
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41863/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 19:08:11.485308 6133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:11.485507 6133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:11.485527 6133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:11.486598 6133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:11.486631 6133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:11.486644 6133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:11.486655 6133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:11.486667 6133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:08:11.486680 6133 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:08:11.486691 6133 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41863/jwks.json
I20260620 19:08:11.488054 6133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:33333
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:33333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41863/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:11.488261 6133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:11.488483 6133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:11.493198 6141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:11.493194 6138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:11.493284 6139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:11.493332 6133 server_base.cc:1034] running on GCE node
I20260620 19:08:11.493746 6133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:11.494031 6133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:11.495183 6133 hybrid_clock.cc:648] HybridClock initialized: now 1781982491495166 us; error 27 us; skew 500 ppm
I20260620 19:08:11.496400 6133 webserver.cc:460] Webserver started at http://127.0.203.62:35399/ using document root <none> and password file <none>
I20260620 19:08:11.496644 6133 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:11.496714 6133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:11.496845 6133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:11.497905 6133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "6000b66665604e19820829081bcb88a5"
format_stamp: "Formatted at 2026-06-20 19:08:11 on dist-test-slave-wd30"
server_key: "2004c60b941237a51571cc236af83066"
server_key_iv: "8866c569df3f4180236e11f4a475544b"
server_key_version: "encryptionkey@0"
I20260620 19:08:11.498281 6133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "6000b66665604e19820829081bcb88a5"
format_stamp: "Formatted at 2026-06-20 19:08:11 on dist-test-slave-wd30"
server_key: "2004c60b941237a51571cc236af83066"
server_key_iv: "8866c569df3f4180236e11f4a475544b"
server_key_version: "encryptionkey@0"
I20260620 19:08:11.499737 6133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:08:11.500464 6147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:11.500643 6133 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:11.500739 6133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "6000b66665604e19820829081bcb88a5"
format_stamp: "Formatted at 2026-06-20 19:08:11 on dist-test-slave-wd30"
server_key: "2004c60b941237a51571cc236af83066"
server_key_iv: "8866c569df3f4180236e11f4a475544b"
server_key_version: "encryptionkey@0"
I20260620 19:08:11.500828 6133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:11.519552 6133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:11.519869 6133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:11.520035 6133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:11.524223 6133 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:33333
I20260620 19:08:11.524255 6199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:33333 every 8 connection(s)
I20260620 19:08:11.524634 6133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:11.525328 6200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:11.528031 6200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5: Bootstrap starting.
I20260620 19:08:11.528632 6200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:11.528927 6200 log.cc:824] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:11.529450 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 6133
I20260620 19:08:11.529525 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:08:11.529704 812 external_mini_cluster.cc:1428] Setting key 0a2eec21be381d8f3f5be60940d21a4c
I20260620 19:08:11.529711 6200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5: No bootstrap required, opened a new log
I20260620 19:08:11.532649 6200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6000b66665604e19820829081bcb88a5" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 33333 } }
I20260620 19:08:11.532816 6200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:11.532846 6200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6000b66665604e19820829081bcb88a5, State: Initialized, Role: FOLLOWER
I20260620 19:08:11.532953 6200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6000b66665604e19820829081bcb88a5" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 33333 } }
I20260620 19:08:11.533023 6200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:11.533061 6200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:11.533126 6200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:11.533660 6200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6000b66665604e19820829081bcb88a5" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 33333 } }
I20260620 19:08:11.533746 6200 leader_election.cc:302] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6000b66665604e19820829081bcb88a5; no voters:
I20260620 19:08:11.533963 6200 leader_election.cc:288] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:08:11.533986 6205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:11.534107 6205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5 [term 1 LEADER]: Becoming Leader. State: Replica: 6000b66665604e19820829081bcb88a5, State: Running, Role: LEADER
I20260620 19:08:11.534192 6205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6000b66665604e19820829081bcb88a5" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 33333 } }
I20260620 19:08:11.534238 6200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:11.534511 6206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6000b66665604e19820829081bcb88a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6000b66665604e19820829081bcb88a5" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 33333 } } }
I20260620 19:08:11.534752 6206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:11.534665 6207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6000b66665604e19820829081bcb88a5. Latest consensus state: current_term: 1 leader_uuid: "6000b66665604e19820829081bcb88a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6000b66665604e19820829081bcb88a5" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 33333 } } }
I20260620 19:08:11.534999 6207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:11.535156 6214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:08:11.535648 6214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:08:11.536398 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:11.530209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51812 (local address 127.0.203.62:33333)
0620 19:08:11.530448 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:11.530457 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:11.530493 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:08:11.531156 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:11.531166 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:11.531245 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:11.531393 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:11.531548 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.533003 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.533811 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.534439 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.534592 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:11.535414 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:11.535421 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:11.535422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:11.535658 (+ 236us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:11.535706 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:11.535723 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:11.535789 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:11.536108 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 19:08:11.536189 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260620 19:08:11.537227 6214 catalog_manager.cc:1347] Generated new cluster ID: 536c861134504981970c096992b20922
I20260620 19:08:11.537284 6214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:08:11.537423 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:08:11.588845 6214 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:08:11.589604 6214 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:11.595593 6214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6000b66665604e19820829081bcb88a5: Generated new TSK 0
I20260620 19:08:11.595798 6214 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:08:11.605316 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:11.601455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51828 (local address 127.0.203.62:33333)
0620 19:08:11.601501 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:11.601504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:11.601576 (+ 72us) server_negotiation.cc:407] Connection header received
0620 19:08:11.601648 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:11.601649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:11.601662 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:11.601750 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:08:11.601874 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.602649 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.603565 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.603888 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.603961 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:11.604117 (+ 156us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:08:11.605097 (+ 980us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41863/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 19:08:11.605234 (+ 137us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51828: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41863/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 19:08:11.605381 6204 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51828: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41863/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 19:08:11.607322 6229 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:11.601357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:33333 (local address 127.0.0.1:51828)
0620 19:08:11.601512 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:11.601523 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:11.601590 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:11.601754 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:11.601755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:11.601757 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:08:11.601815 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:11.601816 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.602755 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.602756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:11.603470 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:11.603472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.603972 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.603973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:11.604019 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:11.604025 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:08:11.605209 (+ 1184us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41863/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 19:08:11.607219 (+ 2010us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.203.62:33333: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41863/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":1767,"thread_start_us":107,"threads_started":1}
I20260620 19:08:11.607672 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 6133
E20260620 19:08:11.616364 6230 webserver.cc:533] Webserver: sslize failed:
E20260620 19:08:11.616441 6230 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T19:08:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (248 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T19:08:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:08:11Z Disabled control of system clock
I20260620 19:08:11.653072 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:36989
--webserver_interface=127.0.203.62
--webserver_port=0
--builtin_ntp_servers=127.0.203.20:39687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:36989
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982468536772-812-0/testchainca.cert with env {}
W20260620 19:08:11.733449 6234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:11.733656 6234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:11.733680 6234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:11.734861 6234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:11.734903 6234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:11.734916 6234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:11.734928 6234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:11.734941 6234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:08:11.734992 6234 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:08:11.735009 6234 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 19:08:11.736539 6234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:39687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:36989
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:36989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982468536772-812-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:11.736794 6234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:11.737015 6234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:08:11.741957 6234 server_base.cc:1034] running on GCE node
W20260620 19:08:11.741914 6240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:11.741950 6239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:11.741930 6242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:11.742439 6234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:11.742779 6234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:11.743969 6234 hybrid_clock.cc:648] HybridClock initialized: now 1781982491743962 us; error 53 us; skew 500 ppm
I20260620 19:08:11.745193 6234 webserver.cc:460] Webserver started at http://127.0.203.62:38907/ using document root <none> and password file <none>
I20260620 19:08:11.745435 6234 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:11.745504 6234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:11.745675 6234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:11.746737 6234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "1590346eddc141919f6163a685b965b1"
format_stamp: "Formatted at 2026-06-20 19:08:11 on dist-test-slave-wd30"
server_key: "963de43f620002e01eac8a1005e65344"
server_key_iv: "e853a2db9381552d1aac2fe5355ccbe2"
server_key_version: "encryptionkey@0"
I20260620 19:08:11.747097 6234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "1590346eddc141919f6163a685b965b1"
format_stamp: "Formatted at 2026-06-20 19:08:11 on dist-test-slave-wd30"
server_key: "963de43f620002e01eac8a1005e65344"
server_key_iv: "e853a2db9381552d1aac2fe5355ccbe2"
server_key_version: "encryptionkey@0"
I20260620 19:08:11.748374 6234 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:11.749066 6248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:11.749250 6234 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:11.749336 6234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "1590346eddc141919f6163a685b965b1"
format_stamp: "Formatted at 2026-06-20 19:08:11 on dist-test-slave-wd30"
server_key: "963de43f620002e01eac8a1005e65344"
server_key_iv: "e853a2db9381552d1aac2fe5355ccbe2"
server_key_version: "encryptionkey@0"
I20260620 19:08:11.749388 6234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:11.778296 6234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:11.778577 6234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:11.778683 6234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:11.782465 6234 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:36989
I20260620 19:08:11.782542 6300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:36989 every 8 connection(s)
I20260620 19:08:11.782897 6234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:11.783895 6301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:11.786370 6301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1: Bootstrap starting.
I20260620 19:08:11.786959 6301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:11.787334 6301 log.cc:824] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:11.787602 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 6234
I20260620 19:08:11.787726 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:08:11.787933 812 external_mini_cluster.cc:1428] Setting key bc17ce15482a28ca3486a03a2fcc796e
I20260620 19:08:11.788182 6301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1: No bootstrap required, opened a new log
I20260620 19:08:11.790305 6301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1590346eddc141919f6163a685b965b1" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36989 } }
I20260620 19:08:11.790463 6301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:11.790508 6301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1590346eddc141919f6163a685b965b1, State: Initialized, Role: FOLLOWER
I20260620 19:08:11.790640 6301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1590346eddc141919f6163a685b965b1" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36989 } }
I20260620 19:08:11.790745 6301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:11.790786 6301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:11.790833 6301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:11.791519 6301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1590346eddc141919f6163a685b965b1" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36989 } }
I20260620 19:08:11.791631 6301 leader_election.cc:302] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1590346eddc141919f6163a685b965b1; no voters:
I20260620 19:08:11.791808 6301 leader_election.cc:288] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:08:11.791862 6306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:11.791999 6306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1 [term 1 LEADER]: Becoming Leader. State: Replica: 1590346eddc141919f6163a685b965b1, State: Running, Role: LEADER
I20260620 19:08:11.792064 6301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:11.792086 6306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1590346eddc141919f6163a685b965b1" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36989 } }
I20260620 19:08:11.792390 6306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1590346eddc141919f6163a685b965b1. Latest consensus state: current_term: 1 leader_uuid: "1590346eddc141919f6163a685b965b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1590346eddc141919f6163a685b965b1" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36989 } } }
I20260620 19:08:11.792485 6306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1 [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:11.792397 6307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1590346eddc141919f6163a685b965b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1590346eddc141919f6163a685b965b1" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36989 } } }
I20260620 19:08:11.792627 6307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1 [sys.catalog]: This master's current role is: LEADER
W20260620 19:08:11.793766 6322 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:08:11.793823 6322 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:08:11.794132 6312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:08:11.794332 6312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:08:11.795068 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:11.788459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51770 (local address 127.0.203.62:36989)
0620 19:08:11.788668 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:11.788676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:11.788700 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:08:11.789440 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:11.789447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:11.789507 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:11.789620 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:11.789861 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.791360 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.792472 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.793145 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.793265 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:11.794297 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:11.794302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:11.794303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:11.794533 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:11.794575 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:11.794589 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:11.794629 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:11.794772 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 19:08:11.794853 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260620 19:08:11.796022 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:08:11.796159 6312 catalog_manager.cc:1347] Generated new cluster ID: db1fa6ecf0a1414587b812f650d0ede7
I20260620 19:08:11.796200 6312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:08:11.807869 6312 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:08:11.808403 6312 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:11.811579 6312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1590346eddc141919f6163a685b965b1: Generated new TSK 0
I20260620 19:08:11.811779 6312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:08:12.245381 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:11.818135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51772 (local address 127.0.203.62:36989)
0620 19:08:11.818180 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:11.818182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:11.818318 (+ 136us) server_negotiation.cc:407] Connection header received
0620 19:08:11.818352 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:11.818352 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:11.818365 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:11.818436 (+ 71us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:08:11.818535 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.819330 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.820088 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.820410 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.820484 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:11.820644 (+ 160us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:08:12.245073 (+424429us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 19:08:12.245296 (+ 223us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51772: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 19:08:12.245466 6330 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:11.818051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36989 (local address 127.0.0.1:51772)
0620 19:08:11.818193 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:11.818204 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:11.818275 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:11.818440 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:11.818441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:11.818443 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:08:11.818491 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:11.818493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.819422 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.819422 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:11.820002 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:11.820003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:11.820497 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:11.820498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:11.820536 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:11.820540 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:08:12.245222 (+424682us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 19:08:12.245352 (+ 130us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.203.62:36989: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":98,"threads_started":1}
W20260620 19:08:12.245474 6305 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51772: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260620 19:08:12.245868 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 6234
2026-06-20T19:08:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (618 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:08:12 dist-test-slave-wd30 krb5kdc[6342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:08:12 dist-test-slave-wd30 krb5kdc[6342](info): set up 2 sockets
Jun 20 19:08:12 dist-test-slave-wd30 krb5kdc[6342](info): commencing operation
krb5kdc: starting...
W20260620 19:08:13.459417 812 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:08:13 dist-test-slave-wd30 krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:08:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:08:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:08:13.578354 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:35485
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:35485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:13.659317 6355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:13.659564 6355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:13.659603 6355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:13.660758 6355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:13.660830 6355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:13.660853 6355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:13.660879 6355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:13.660910 6355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:08:13.662423 6355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:35485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:35485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:13.662714 6355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:13.662951 6355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:13.667829 6360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:13.667827 6363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:13.667917 6355 server_base.cc:1034] running on GCE node
W20260620 19:08:13.667836 6361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:13.668294 6355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:13.668671 6355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:13.669874 6355 hybrid_clock.cc:648] HybridClock initialized: now 1781982493669859 us; error 30 us; skew 500 ppm
Jun 20 19:08:13 dist-test-slave-wd30 krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:13.672585 6355 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:08:13.673077 6355 webserver.cc:460] Webserver started at http://127.0.203.62:33775/ using document root <none> and password file <none>
I20260620 19:08:13.673307 6355 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:13.673384 6355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:13.673533 6355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:13.674597 6355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "d135335a48994dbebb44c6429a74db93"
format_stamp: "Formatted at 2026-06-20 19:08:13 on dist-test-slave-wd30"
server_key: "d2325537f9a9d216e194f2119fea5ef1"
server_key_iv: "7106237b96c60c21e754d52dd3af9782"
server_key_version: "encryptionkey@0"
I20260620 19:08:13.674965 6355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "d135335a48994dbebb44c6429a74db93"
format_stamp: "Formatted at 2026-06-20 19:08:13 on dist-test-slave-wd30"
server_key: "d2325537f9a9d216e194f2119fea5ef1"
server_key_iv: "7106237b96c60c21e754d52dd3af9782"
server_key_version: "encryptionkey@0"
I20260620 19:08:13.676327 6355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:08:13.677040 6370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:13.677213 6355 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:08:13.677310 6355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "d135335a48994dbebb44c6429a74db93"
format_stamp: "Formatted at 2026-06-20 19:08:13 on dist-test-slave-wd30"
server_key: "d2325537f9a9d216e194f2119fea5ef1"
server_key_iv: "7106237b96c60c21e754d52dd3af9782"
server_key_version: "encryptionkey@0"
I20260620 19:08:13.677403 6355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:13.688968 6355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:13.690655 6355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:13.690814 6355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:13.694836 6355 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:35485
I20260620 19:08:13.694866 6422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:35485 every 8 connection(s)
I20260620 19:08:13.695384 6355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:13.696007 6423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:13.698138 6423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93: Bootstrap starting.
I20260620 19:08:13.698647 6423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:13.698886 6423 log.cc:824] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:13.699582 6423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93: No bootstrap required, opened a new log
I20260620 19:08:13.701233 6423 raft_consensus.cc:348] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d135335a48994dbebb44c6429a74db93" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35485 } }
I20260620 19:08:13.701367 6423 raft_consensus.cc:374] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:13.701390 6423 raft_consensus.cc:729] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d135335a48994dbebb44c6429a74db93, State: Initialized, Role: FOLLOWER
I20260620 19:08:13.701522 6423 consensus_queue.cc:260] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d135335a48994dbebb44c6429a74db93" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35485 } }
I20260620 19:08:13.701612 6423 raft_consensus.cc:388] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:13.701635 6423 raft_consensus.cc:482] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:13.701663 6423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:13.702481 6423 raft_consensus.cc:504] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d135335a48994dbebb44c6429a74db93" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35485 } }
I20260620 19:08:13.702564 6423 leader_election.cc:302] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d135335a48994dbebb44c6429a74db93; no voters:
I20260620 19:08:13.702674 6423 leader_election.cc:288] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:08:13.703003 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 6355
I20260620 19:08:13.703097 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:08:13.703290 812 external_mini_cluster.cc:1428] Setting key f8187f1dd383f83ccbbed83bb5c074db
I20260620 19:08:13.702754 6426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:13.702903 6423 sys_catalog.cc:564] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:13.703593 6426 raft_consensus.cc:686] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93 [term 1 LEADER]: Becoming Leader. State: Replica: d135335a48994dbebb44c6429a74db93, State: Running, Role: LEADER
I20260620 19:08:13.703693 6426 consensus_queue.cc:237] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d135335a48994dbebb44c6429a74db93" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35485 } }
I20260620 19:08:13.704100 6428 sys_catalog.cc:455] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d135335a48994dbebb44c6429a74db93. Latest consensus state: current_term: 1 leader_uuid: "d135335a48994dbebb44c6429a74db93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d135335a48994dbebb44c6429a74db93" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35485 } } }
I20260620 19:08:13.704185 6428 sys_catalog.cc:458] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93 [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:13.704623 6437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:08:13.704694 6427 sys_catalog.cc:455] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d135335a48994dbebb44c6429a74db93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d135335a48994dbebb44c6429a74db93" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35485 } } }
I20260620 19:08:13.704752 6427 sys_catalog.cc:458] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93 [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:13.705168 6437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:08:13.706734 6437 catalog_manager.cc:1347] Generated new cluster ID: 544509bcd3f9472eae6006bd226e2fd5
I20260620 19:08:13.706782 6437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:08:13 dist-test-slave-wd30 krb5kdc[6342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:13.714360 6437 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:08:13.715154 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:13.703919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.0.203.62:35485)
0620 19:08:13.704107 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:13.704119 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:13.704154 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:08:13.704408 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:13.704416 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:13.704473 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:13.705205 (+ 732us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:13.705459 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:13.706914 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:13.707985 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:13.708623 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:13.708724 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:13.711507 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:13.711512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:13.711512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:13.711525 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:13.713507 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:13.713886 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:13.713890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:13.713891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:13.713986 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:13.714196 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:13.714197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:13.714197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:13.714487 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:13.714659 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:13.714758 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 19:08:13.714911 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":71,"threads_started":1}
I20260620 19:08:13.715168 6437 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:13.721360 6437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d135335a48994dbebb44c6429a74db93: Generated new TSK 0
I20260620 19:08:13.721562 6437 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
I20260620 19:08:13.751590 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:35485
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:13.835213 6449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:13.835413 6449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:13.835438 6449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:13.836712 6449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:13.836752 6449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:13.836778 6449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:08:13.838229 6449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:35485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:13.838480 6449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:13.838721 6449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:13.843672 6457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:13.843657 6454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:13.843828 6449 server_base.cc:1034] running on GCE node
W20260620 19:08:13.843719 6455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:13.844162 6449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:13.844411 6449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:13.845574 6449 hybrid_clock.cc:648] HybridClock initialized: now 1781982493845562 us; error 31 us; skew 500 ppm
Jun 20 19:08:13 dist-test-slave-wd30 krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:13.848089 6449 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:08:13.848498 6449 webserver.cc:460] Webserver started at http://127.0.203.1:37405/ using document root <none> and password file <none>
I20260620 19:08:13.848742 6449 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:13.848807 6449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:13.848960 6449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:13.849972 6449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "43d0f4bf9d6241eba95a79ab075e7f3c"
format_stamp: "Formatted at 2026-06-20 19:08:13 on dist-test-slave-wd30"
server_key: "5df59d55a87589a448307bebc9ef0da8"
server_key_iv: "f92efee5bfbaf998095131e1a5809781"
server_key_version: "encryptionkey@0"
I20260620 19:08:13.850325 6449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "43d0f4bf9d6241eba95a79ab075e7f3c"
format_stamp: "Formatted at 2026-06-20 19:08:13 on dist-test-slave-wd30"
server_key: "5df59d55a87589a448307bebc9ef0da8"
server_key_iv: "f92efee5bfbaf998095131e1a5809781"
server_key_version: "encryptionkey@0"
I20260620 19:08:13.851684 6449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 19:08:13.852375 6464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:13.852547 6449 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:13.852650 6449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "43d0f4bf9d6241eba95a79ab075e7f3c"
format_stamp: "Formatted at 2026-06-20 19:08:13 on dist-test-slave-wd30"
server_key: "5df59d55a87589a448307bebc9ef0da8"
server_key_iv: "f92efee5bfbaf998095131e1a5809781"
server_key_version: "encryptionkey@0"
I20260620 19:08:13.852727 6449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:13.869590 6449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:13.871654 6449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:13.871814 6449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:13.872048 6449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:13.872344 6449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:13.872398 6449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:13.872444 6449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:13.872473 6449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:13.878147 6449 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:37727
I20260620 19:08:13.878219 6577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:37727 every 8 connection(s)
I20260620 19:08:13.878533 6449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:08:13 dist-test-slave-wd30 krb5kdc[6342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:13.886831 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 6449
I20260620 19:08:13.886966 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:08:13.887208 812 external_mini_cluster.cc:1428] Setting key 77dfb77f825fa38e621a51c1e3c52782
I20260620 19:08:13.888123 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:13.879302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:60385 (local address 127.0.203.62:35485)
0620 19:08:13.879366 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:13.879368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:13.879583 (+ 215us) server_negotiation.cc:407] Connection header received
0620 19:08:13.880643 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:13.880645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:13.880660 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:13.880725 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:13.881406 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:13.881926 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:13.882957 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:13.883247 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:13.883333 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:13.885296 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:13.885298 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:13.885298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:13.885305 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:13.886473 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:13.887050 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:13.887051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:13.887052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:13.887159 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:13.887455 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:13.887456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:13.887456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:13.887611 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:13.887680 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:13.887977 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 19:08:13.888060 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:08:13.888386 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:13.879300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:35485 (local address 127.0.203.1:60385)
0620 19:08:13.879463 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:13.879497 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:13.880535 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:13.880795 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:13.880799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:13.881115 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:13.881306 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:13.881310 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:13.882025 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:13.882026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:13.882858 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:13.882860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:13.883343 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:13.883344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:13.883471 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:13.883884 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:13.883893 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:13.885186 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:13.886573 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:13.886580 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:13.886582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:13.886908 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:13.887220 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:13.887221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:13.887222 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:13.887361 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:13.887681 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:13.887687 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:13.887830 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 19:08:13.887942 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 19:08:13.888113 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260620 19:08:13.888870 6578 heartbeater.cc:347] Connected to a master server at 127.0.203.62:35485
I20260620 19:08:13.888981 6578 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:13.889155 6578 heartbeater.cc:511] Master 127.0.203.62:35485 requested a full tablet report, sending...
I20260620 19:08:13.889745 6387 ts_manager.cc:194] Registered new tserver with Master: 43d0f4bf9d6241eba95a79ab075e7f3c (127.0.203.1:37727)
I20260620 19:08:13.890592 6387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:60385
WARNING: no policy specified for kudu/127.0.203.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
I20260620 19:08:13.911844 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:35485
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:13.993184 6583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:13.993386 6583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:13.993417 6583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:13.994514 6583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:13.994545 6583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:13.994571 6583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:08:13.996042 6583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:35485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:13.996230 6583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:13.996397 6583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:14.001224 6588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:14.001240 6591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:14.001442 6589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:14.001608 6583 server_base.cc:1034] running on GCE node
I20260620 19:08:14.001842 6583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:14.002039 6583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:14.003176 6583 hybrid_clock.cc:648] HybridClock initialized: now 1781982494003163 us; error 26 us; skew 500 ppm
Jun 20 19:08:14 dist-test-slave-wd30 krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:14.005704 6583 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:08:14.006166 6583 webserver.cc:460] Webserver started at http://127.0.203.2:37833/ using document root <none> and password file <none>
I20260620 19:08:14.006453 6583 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:14.006498 6583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:14.006645 6583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:14.007746 6583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "a38d466b2ec046a98e989e62c9f5f6b8"
format_stamp: "Formatted at 2026-06-20 19:08:14 on dist-test-slave-wd30"
server_key: "af7cad449edb35e2403276648ed81ce0"
server_key_iv: "c958c77e4dc07c1cdfa9b53454885e82"
server_key_version: "encryptionkey@0"
I20260620 19:08:14.008103 6583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "a38d466b2ec046a98e989e62c9f5f6b8"
format_stamp: "Formatted at 2026-06-20 19:08:14 on dist-test-slave-wd30"
server_key: "af7cad449edb35e2403276648ed81ce0"
server_key_iv: "c958c77e4dc07c1cdfa9b53454885e82"
server_key_version: "encryptionkey@0"
I20260620 19:08:14.009399 6583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:08:14.010126 6598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:14.010293 6583 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:14.010401 6583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "a38d466b2ec046a98e989e62c9f5f6b8"
format_stamp: "Formatted at 2026-06-20 19:08:14 on dist-test-slave-wd30"
server_key: "af7cad449edb35e2403276648ed81ce0"
server_key_iv: "c958c77e4dc07c1cdfa9b53454885e82"
server_key_version: "encryptionkey@0"
I20260620 19:08:14.010485 6583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:14.029325 6583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:14.031141 6583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:14.031311 6583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:14.031551 6583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:14.031857 6583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:14.031911 6583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:14.031947 6583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:14.031996 6583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:14.037746 6583 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:40301
I20260620 19:08:14.037796 6711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:40301 every 8 connection(s)
I20260620 19:08:14.038108 6583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:08:14 dist-test-slave-wd30 krb5kdc[6342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:14.046694 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 6583
I20260620 19:08:14.046818 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:08:14.047083 812 external_mini_cluster.cc:1428] Setting key 8556876eb4f11fc86a185c4ea4f236ca
I20260620 19:08:14.047993 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:14.038842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:54109 (local address 127.0.203.62:35485)
0620 19:08:14.038890 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:14.038892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:14.039103 (+ 211us) server_negotiation.cc:407] Connection header received
0620 19:08:14.040133 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:14.040135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:14.040147 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:14.040206 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:14.040835 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:14.041386 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:14.042426 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:14.042623 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:14.042702 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:14.044858 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:14.044859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:14.044860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:14.044867 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:14.046128 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:14.046857 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:14.046859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:14.046859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:14.046927 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:14.047250 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:14.047251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:14.047251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:14.047415 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:14.047489 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:14.047784 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 19:08:14.047887 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:08:14.048089 6714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:14.038804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:35485 (local address 127.0.203.2:54109)
0620 19:08:14.038983 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:14.039019 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:14.039983 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:14.040278 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:14.040282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:14.040549 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:14.040735 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:14.040739 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:14.041475 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:14.041476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:14.042297 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:14.042298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:14.042705 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:14.042706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:14.042844 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:14.043403 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:14.043411 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:14.044743 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:14.046196 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:14.046202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:14.046204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:14.046513 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:14.047022 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:14.047023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:14.047023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:14.047173 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:14.047491 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:14.047498 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:14.047631 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 19:08:14.047742 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 19:08:14.047902 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260620 19:08:14.048570 6712 heartbeater.cc:347] Connected to a master server at 127.0.203.62:35485
I20260620 19:08:14.048673 6712 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:14.048857 6712 heartbeater.cc:511] Master 127.0.203.62:35485 requested a full tablet report, sending...
I20260620 19:08:14.049259 6387 ts_manager.cc:194] Registered new tserver with Master: a38d466b2ec046a98e989e62c9f5f6b8 (127.0.203.2:40301)
I20260620 19:08:14.049647 6387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:54109
WARNING: no policy specified for kudu/127.0.203.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
I20260620 19:08:14.071659 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:35485
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:14.152544 6717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:14.152808 6717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:14.152858 6717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:14.154064 6717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:14.154142 6717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:14.154189 6717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:08:14.155827 6717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:35485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:14.156098 6717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:14.156337 6717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:14.161593 6725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:14.161602 6722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:14.161590 6723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:14.161619 6717 server_base.cc:1034] running on GCE node
I20260620 19:08:14.162035 6717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:14.162312 6717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:14.163482 6717 hybrid_clock.cc:648] HybridClock initialized: now 1781982494163463 us; error 52 us; skew 500 ppm
Jun 20 19:08:14 dist-test-slave-wd30 krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:14.165974 6717 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:08:14.166442 6717 webserver.cc:460] Webserver started at http://127.0.203.3:45223/ using document root <none> and password file <none>
I20260620 19:08:14.166625 6717 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:14.166666 6717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:14.166754 6717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:14.167889 6717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "a8b647c4251d469badc133fe16a1609e"
format_stamp: "Formatted at 2026-06-20 19:08:14 on dist-test-slave-wd30"
server_key: "f43793bb6201e14c3a11603b5e3b583b"
server_key_iv: "759fd0e42bcaeeb87f017f8f4f8f2dde"
server_key_version: "encryptionkey@0"
I20260620 19:08:14.168198 6717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "a8b647c4251d469badc133fe16a1609e"
format_stamp: "Formatted at 2026-06-20 19:08:14 on dist-test-slave-wd30"
server_key: "f43793bb6201e14c3a11603b5e3b583b"
server_key_iv: "759fd0e42bcaeeb87f017f8f4f8f2dde"
server_key_version: "encryptionkey@0"
I20260620 19:08:14.169508 6717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:08:14.170264 6732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:14.170457 6717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:14.170542 6717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "a8b647c4251d469badc133fe16a1609e"
format_stamp: "Formatted at 2026-06-20 19:08:14 on dist-test-slave-wd30"
server_key: "f43793bb6201e14c3a11603b5e3b583b"
server_key_iv: "759fd0e42bcaeeb87f017f8f4f8f2dde"
server_key_version: "encryptionkey@0"
I20260620 19:08:14.170591 6717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:14.186105 6717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:14.188174 6717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:14.188283 6717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:14.188480 6717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:14.188766 6717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:14.188798 6717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:14.188822 6717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:14.188835 6717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:14.194278 6717 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:38465
I20260620 19:08:14.194355 6845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:38465 every 8 connection(s)
I20260620 19:08:14.194656 6717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:08:14.196278 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 6717
I20260620 19:08:14.196352 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:08:14.196544 812 external_mini_cluster.cc:1428] Setting key de1db991482bcb66103b4a1174117211
Jun 20 19:08:14 dist-test-slave-wd30 krb5kdc[6342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:14.204067 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:14.195459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:47285 (local address 127.0.203.62:35485)
0620 19:08:14.195525 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:14.195527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:14.195733 (+ 206us) server_negotiation.cc:407] Connection header received
0620 19:08:14.196808 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:14.196810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:14.196821 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:14.196909 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:14.197702 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:14.198245 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:14.199196 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:14.199383 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:14.199430 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:14.201490 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:14.201491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:14.201491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:14.201498 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:14.202711 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:14.203104 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:14.203105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:14.203105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:14.203191 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:14.203403 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:14.203404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:14.203404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:14.203501 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:14.203562 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:14.203877 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 19:08:14.204003 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:08:14.204386 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:14.195443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:35485 (local address 127.0.203.3:47285)
0620 19:08:14.195615 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:14.195650 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:14.196698 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:14.197029 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:14.197033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:14.197396 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:14.197615 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:14.197620 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:14.198322 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:14.198322 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:14.199111 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:14.199113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:14.199458 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:14.199459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:14.199599 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:14.200112 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:14.200120 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:14.201408 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:14.202782 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:14.202787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:14.202789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:14.203015 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:14.203249 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:14.203250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:14.203250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:14.203345 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:14.203596 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:14.203603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:14.203726 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 19:08:14.203861 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 19:08:14.204064 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20260620 19:08:14.204924 6846 heartbeater.cc:347] Connected to a master server at 127.0.203.62:35485
I20260620 19:08:14.205047 6846 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:14.205228 6846 heartbeater.cc:511] Master 127.0.203.62:35485 requested a full tablet report, sending...
I20260620 19:08:14.205621 6387 ts_manager.cc:194] Registered new tserver with Master: a8b647c4251d469badc133fe16a1609e (127.0.203.3:38465)
I20260620 19:08:14.205998 6387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:47285
I20260620 19:08:14.208178 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/test-admin.keytab.
Jun 20 19:08:14 dist-test-slave-wd30 krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:14.224045 812 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 19:08:14.225924 6853 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:14.225083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33948 (local address 127.0.203.1:37727)
0620 19:08:14.225256 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:14.225260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:14.225270 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:08:14.225295 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:14.225297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:14.225310 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:14.225408 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:14.225831 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33948: BlockingRecv error: recv got EOF from 127.0.0.1:33948 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
W20260620 19:08:14.226245 812 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/krb5cc' of type FILE
Jun 20 19:08:14 dist-test-slave-wd30 krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:14.227429 812 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 19:08:14 dist-test-slave-wd30 krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:14.228371 812 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 19:08:14 dist-test-slave-wd30 krb5kdc[6342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.1@KRBTEST.COM
I20260620 19:08:14.237339 6853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:14.229004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.0.203.1:37727)
0620 19:08:14.229070 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:14.229073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:14.229144 (+ 71us) server_negotiation.cc:407] Connection header received
0620 19:08:14.229227 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:14.229228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:14.229245 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:14.229308 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:14.229758 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:14.230543 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:14.231450 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:14.231795 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:14.231887 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:14.234196 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:14.234206 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:14.234206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:14.234214 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:14.235997 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:14.236417 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:14.236420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:14.236421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:14.236483 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:14.236703 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:14.236705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:14.236705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:14.236866 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:14.237075 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:14.237124 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 19:08:14.237207 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 19:08:14.239063 6853 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:14.238579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33958 (local address 127.0.203.1:37727)
0620 19:08:14.238616 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:14.238618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:14.238676 (+ 58us) server_negotiation.cc:407] Connection header received
0620 19:08:14.238717 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:14.238717 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:14.238727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:14.238777 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:14.239003 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33958: BlockingRecv error: recv got EOF from 127.0.0.1:33958 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 19:08:14.239238 812 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/krb5cc' of type FILE
Jun 20 19:08:14 dist-test-slave-wd30 krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:14.240284 812 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 19:08:14 dist-test-slave-wd30 krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:14.241245 812 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 19:08:14 dist-test-slave-wd30 krb5kdc[6342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.1@KRBTEST.COM
I20260620 19:08:14.248764 6853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:14.241785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33970 (local address 127.0.203.1:37727)
0620 19:08:14.241823 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:14.241826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:14.241931 (+ 105us) server_negotiation.cc:407] Connection header received
0620 19:08:14.241999 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:14.242000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:14.242011 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:14.242087 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:14.242548 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:14.243111 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:14.243934 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:14.244162 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:14.244255 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:14.246321 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:14.246322 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:14.246323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:14.246329 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:14.247778 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:14.248105 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:14.248106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:14.248107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:14.248154 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:14.248313 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:14.248313 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:14.248314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:14.248397 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:14.248468 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:14.248556 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 19:08:14.248624 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 19:08:14.250241 6853 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:14.249692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.0.203.1:37727)
0620 19:08:14.249743 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:14.249745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:14.249758 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:08:14.249808 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:14.249809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:14.249817 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:14.249869 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:14.250171 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33974: BlockingRecv error: recv got EOF from 127.0.0.1:33974 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 19:08:14.250334 812 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982468536772-812-0/krb5kdc/krb5cc' of type FILE
Jun 20 19:08:14 dist-test-slave-wd30 krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:14.251331 812 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 19:08:14 dist-test-slave-wd30 krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:14.252128 812 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 19:08:14 dist-test-slave-wd30 krb5kdc[6342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.1@KRBTEST.COM
I20260620 19:08:14.258972 6853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:14.252672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33982 (local address 127.0.203.1:37727)
0620 19:08:14.252726 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:14.252728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:14.252737 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:08:14.252829 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:14.252830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:14.252840 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:14.252932 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:14.253299 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:14.253757 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:14.254452 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:14.254644 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:14.254723 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:14.256883 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:14.256884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:14.256884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:14.256890 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:14.258108 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:14.258395 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:14.258396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:14.258396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:14.258443 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:14.258609 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:14.258610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:14.258610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:14.258692 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:14.258757 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:14.258864 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 19:08:14.258916 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:08:14.259725 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 6449
I20260620 19:08:14.265329 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 6583
I20260620 19:08:14.269671 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 6717
I20260620 19:08:14.273734 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 6355
2026-06-20T19:08:14Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1852 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:08:14 dist-test-slave-wd30 krb5kdc[6866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:08:14 dist-test-slave-wd30 krb5kdc[6866](info): set up 2 sockets
Jun 20 19:08:14 dist-test-slave-wd30 krb5kdc[6866](info): commencing operation
krb5kdc: starting...
W20260620 19:08:15.311379 812 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:08:15 dist-test-slave-wd30 krb5kdc[6866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:08:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:08:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.62.keytab.
Entry for principal oryx/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.62.keytab.
I20260620 19:08:15.429616 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:36095
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.62.keytab
--principal=oryx/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:36095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:15.511036 6879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:15.511274 6879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:15.511299 6879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:15.512523 6879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:15.512567 6879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:15.512581 6879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:15.512601 6879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:15.512656 6879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:08:15.514237 6879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:36095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.62.keytab
--principal=oryx/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:36095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:15.514495 6879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:15.514739 6879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:15.519424 6887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:15.519495 6885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:15.519544 6879 server_base.cc:1034] running on GCE node
W20260620 19:08:15.519428 6884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:15.519822 6879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:15.520108 6879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:15.521238 6879 hybrid_clock.cc:648] HybridClock initialized: now 1781982495521228 us; error 29 us; skew 500 ppm
Jun 20 19:08:15 dist-test-slave-wd30 krb5kdc[6866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982495, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:15.527088 6879 init.cc:377] Logged in from keytab as oryx/127.0.203.62@KRBTEST.COM (short username oryx)
I20260620 19:08:15.527652 6879 webserver.cc:460] Webserver started at http://127.0.203.62:45553/ using document root <none> and password file <none>
I20260620 19:08:15.527913 6879 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:15.527959 6879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:15.528127 6879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:15.529232 6879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "b58adcd02d204af8810c43fa357d0ffa"
format_stamp: "Formatted at 2026-06-20 19:08:15 on dist-test-slave-wd30"
server_key: "eb0d480ceb3747604a624ce3f106aca1"
server_key_iv: "a1701f968a441114b82261b0ac0979c6"
server_key_version: "encryptionkey@0"
I20260620 19:08:15.529603 6879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "b58adcd02d204af8810c43fa357d0ffa"
format_stamp: "Formatted at 2026-06-20 19:08:15 on dist-test-slave-wd30"
server_key: "eb0d480ceb3747604a624ce3f106aca1"
server_key_iv: "a1701f968a441114b82261b0ac0979c6"
server_key_version: "encryptionkey@0"
I20260620 19:08:15.530953 6879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:08:15.531764 6894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:15.531934 6879 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:15.532033 6879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "b58adcd02d204af8810c43fa357d0ffa"
format_stamp: "Formatted at 2026-06-20 19:08:15 on dist-test-slave-wd30"
server_key: "eb0d480ceb3747604a624ce3f106aca1"
server_key_iv: "a1701f968a441114b82261b0ac0979c6"
server_key_version: "encryptionkey@0"
I20260620 19:08:15.532116 6879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:15.549623 6879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:15.551769 6879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:15.551932 6879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:15.556120 6879 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:36095
I20260620 19:08:15.556182 6946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:36095 every 8 connection(s)
I20260620 19:08:15.556555 6879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:15.557515 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:15.559760 6947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa: Bootstrap starting.
I20260620 19:08:15.560303 6947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:15.560532 6947 log.cc:824] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:15.561152 6947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa: No bootstrap required, opened a new log
I20260620 19:08:15.562670 6947 raft_consensus.cc:348] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } }
I20260620 19:08:15.562806 6947 raft_consensus.cc:374] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:15.562830 6947 raft_consensus.cc:729] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b58adcd02d204af8810c43fa357d0ffa, State: Initialized, Role: FOLLOWER
I20260620 19:08:15.562980 6947 consensus_queue.cc:260] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } }
I20260620 19:08:15.563067 6947 raft_consensus.cc:388] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:15.563092 6947 raft_consensus.cc:482] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:15.563185 6947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:15.563948 6947 raft_consensus.cc:504] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } }
I20260620 19:08:15.564033 6947 leader_election.cc:302] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b58adcd02d204af8810c43fa357d0ffa; no voters:
I20260620 19:08:15.564143 6947 leader_election.cc:288] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:08:15.564193 6950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:15.564368 6950 raft_consensus.cc:686] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [term 1 LEADER]: Becoming Leader. State: Replica: b58adcd02d204af8810c43fa357d0ffa, State: Running, Role: LEADER
I20260620 19:08:15.564414 6947 sys_catalog.cc:564] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:15.564452 6950 consensus_queue.cc:237] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } }
I20260620 19:08:15.564852 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 6879
I20260620 19:08:15.564982 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:08:15.565200 812 external_mini_cluster.cc:1428] Setting key c1276226c11d6d4a604866c9db2c868b
I20260620 19:08:15.565650 6951 sys_catalog.cc:455] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b58adcd02d204af8810c43fa357d0ffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } }
I20260620 19:08:15.565769 6951 sys_catalog.cc:458] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:15.566041 6952 sys_catalog.cc:455] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: SysCatalogTable state changed. Reason: New leader b58adcd02d204af8810c43fa357d0ffa. Latest consensus state: current_term: 1 leader_uuid: "b58adcd02d204af8810c43fa357d0ffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } }
I20260620 19:08:15.566116 6952 sys_catalog.cc:458] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: This master's current role is: LEADER
W20260620 19:08:15.566345 6967 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:08:15.566412 6967 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:08:15.566480 6968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:08:15.566659 6968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:08:15.568614 6968 catalog_manager.cc:1347] Generated new cluster ID: b794508261304e84b9d0314b8f040e46
I20260620 19:08:15.568679 6968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:08:15 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM
I20260620 19:08:15.576820 6966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:15.565659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44226 (local address 127.0.203.62:36095)
0620 19:08:15.565827 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:15.565837 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:15.566117 (+ 280us) server_negotiation.cc:407] Connection header received
0620 19:08:15.566256 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:15.566264 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:15.566325 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:15.566468 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:15.567310 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:15.568505 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:15.569576 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:15.570165 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:15.570263 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:15.573048 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:15.573053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:15.573054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:15.573069 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:15.575172 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:15.575558 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:15.575561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:15.575561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:15.575686 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:15.575893 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:15.575894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:15.575894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:15.576217 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:15.576380 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:15.576460 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 19:08:15.576602 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260620 19:08:15.580191 6968 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:08:15.580667 6968 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:15.590412 6968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa: Generated new TSK 0
I20260620 19:08:15.590617 6968 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.1.keytab.
Entry for principal oryx/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.1.keytab.
I20260620 19:08:15.617461 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36095
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.1.keytab
--principal=oryx/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:15.699434 6973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:15.699637 6973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:15.699659 6973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:15.700723 6973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:15.700760 6973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:15.700788 6973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:08:15.702219 6973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.1.keytab
--principal=oryx/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:15.702392 6973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:15.702558 6973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:15.707652 6981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:15.707669 6979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:15.707664 6978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:15.707726 6973 server_base.cc:1034] running on GCE node
I20260620 19:08:15.708104 6973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:15.708374 6973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:15.709513 6973 hybrid_clock.cc:648] HybridClock initialized: now 1781982495709501 us; error 31 us; skew 500 ppm
Jun 20 19:08:15 dist-test-slave-wd30 krb5kdc[6866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982495, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:15.712061 6973 init.cc:377] Logged in from keytab as oryx/127.0.203.1@KRBTEST.COM (short username oryx)
I20260620 19:08:15.712472 6973 webserver.cc:460] Webserver started at http://127.0.203.1:41453/ using document root <none> and password file <none>
I20260620 19:08:15.712687 6973 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:15.712738 6973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:15.712878 6973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:15.713896 6973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "6a241202a70f420ca87abfce500521fa"
format_stamp: "Formatted at 2026-06-20 19:08:15 on dist-test-slave-wd30"
server_key: "1ce971251b0292d83ebf101c3e097d02"
server_key_iv: "30e4ce31c71a9d1b6a524de80df26589"
server_key_version: "encryptionkey@0"
I20260620 19:08:15.714251 6973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "6a241202a70f420ca87abfce500521fa"
format_stamp: "Formatted at 2026-06-20 19:08:15 on dist-test-slave-wd30"
server_key: "1ce971251b0292d83ebf101c3e097d02"
server_key_iv: "30e4ce31c71a9d1b6a524de80df26589"
server_key_version: "encryptionkey@0"
I20260620 19:08:15.715785 6973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:08:15.716550 6988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:15.716853 6973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:08:15.716943 6973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "6a241202a70f420ca87abfce500521fa"
format_stamp: "Formatted at 2026-06-20 19:08:15 on dist-test-slave-wd30"
server_key: "1ce971251b0292d83ebf101c3e097d02"
server_key_iv: "30e4ce31c71a9d1b6a524de80df26589"
server_key_version: "encryptionkey@0"
I20260620 19:08:15.717015 6973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:15.723626 6973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:15.725386 6973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:15.725536 6973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:15.725750 6973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:15.726049 6973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:15.726104 6973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:15.726143 6973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:15.726171 6973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:15.731745 6973 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:40925
I20260620 19:08:15.731835 7101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:40925 every 8 connection(s)
I20260620 19:08:15.732113 6973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:08:15.732438 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 6973
I20260620 19:08:15.732543 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:08:15.732777 812 external_mini_cluster.cc:1428] Setting key 36c35b0f3128b8f214953a3614235728
Jun 20 19:08:15 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982495, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.203.2@KRBTEST.COM; defaulting to no policy
I20260620 19:08:15.743180 6966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:15.732866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:59345 (local address 127.0.203.62:36095)
0620 19:08:15.732923 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:15.732925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:15.733272 (+ 347us) server_negotiation.cc:407] Connection header received
0620 19:08:15.733857 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:15.733858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:15.733870 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:15.733925 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:15.735159 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:15.735742 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:15.737314 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:15.737512 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:15.737577 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:15.740183 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:15.740185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:15.740185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:15.740195 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:15.741641 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:15.742155 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:15.742157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:15.742157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:15.742227 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:15.742482 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:15.742483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:15.742484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:15.742610 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:15.742690 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:15.742978 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 19:08:15.743061 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.2@KRBTEST.COM" created.
I20260620 19:08:15.743388 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:15.732801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.1:59345)
0620 19:08:15.733157 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:15.733186 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:15.733754 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:15.733997 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:15.734000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:15.734703 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:15.735020 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:15.735028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:15.735837 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:15.735838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:15.737208 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:15.737211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:15.737610 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:15.737612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:15.737774 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:15.738309 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:15.738320 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:15.740016 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:15.741745 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:15.741752 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:15.741754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:15.742075 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:15.742297 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:15.742298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:15.742298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:15.742415 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:15.742681 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:15.742687 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:15.742829 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 19:08:15.742953 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 19:08:15.743144 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":173,"threads_started":1}
I20260620 19:08:15.744091 7102 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36095
I20260620 19:08:15.744210 7102 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:15.744385 7102 heartbeater.cc:511] Master 127.0.203.62:36095 requested a full tablet report, sending...
I20260620 19:08:15.745010 6911 ts_manager.cc:194] Registered new tserver with Master: 6a241202a70f420ca87abfce500521fa (127.0.203.1:40925)
I20260620 19:08:15.745666 6911 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:59345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.2.keytab.
Entry for principal oryx/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.2.keytab.
I20260620 19:08:15.757395 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36095
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.2.keytab
--principal=oryx/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:15.838701 7107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:15.838910 7107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:15.838934 7107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:15.840169 7107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:15.840214 7107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:15.840240 7107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:08:15.841672 7107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.2.keytab
--principal=oryx/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:15.841912 7107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:15.842162 7107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:15.847009 7113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:15.847011 7112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:15.847051 7115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:15.847084 7107 server_base.cc:1034] running on GCE node
I20260620 19:08:15.847470 7107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:15.847715 7107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:15.848879 7107 hybrid_clock.cc:648] HybridClock initialized: now 1781982495848864 us; error 35 us; skew 500 ppm
Jun 20 19:08:15 dist-test-slave-wd30 krb5kdc[6866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982495, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:15.851318 7107 init.cc:377] Logged in from keytab as oryx/127.0.203.2@KRBTEST.COM (short username oryx)
I20260620 19:08:15.851801 7107 webserver.cc:460] Webserver started at http://127.0.203.2:39119/ using document root <none> and password file <none>
I20260620 19:08:15.852037 7107 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:15.852106 7107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:15.852246 7107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:15.853279 7107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "73d53fd8b46644c4a48293779879f814"
format_stamp: "Formatted at 2026-06-20 19:08:15 on dist-test-slave-wd30"
server_key: "0123ea4c3d356d53c14a29d4dabf0ddf"
server_key_iv: "34bb0be9e3e36090e9dde65d340d917a"
server_key_version: "encryptionkey@0"
I20260620 19:08:15.853639 7107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "73d53fd8b46644c4a48293779879f814"
format_stamp: "Formatted at 2026-06-20 19:08:15 on dist-test-slave-wd30"
server_key: "0123ea4c3d356d53c14a29d4dabf0ddf"
server_key_iv: "34bb0be9e3e36090e9dde65d340d917a"
server_key_version: "encryptionkey@0"
I20260620 19:08:15.854967 7107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:08:15.855800 7122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:15.855996 7107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:08:15.856094 7107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "73d53fd8b46644c4a48293779879f814"
format_stamp: "Formatted at 2026-06-20 19:08:15 on dist-test-slave-wd30"
server_key: "0123ea4c3d356d53c14a29d4dabf0ddf"
server_key_iv: "34bb0be9e3e36090e9dde65d340d917a"
server_key_version: "encryptionkey@0"
I20260620 19:08:15.856226 7107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:15.864444 7107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:15.866513 7107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:15.866668 7107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:15.866880 7107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:15.867247 7107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:15.867298 7107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:15.867337 7107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:15.867362 7107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:15.873395 7107 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:42711
I20260620 19:08:15.873488 7235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:42711 every 8 connection(s)
I20260620 19:08:15.873790 7107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:08:15 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982495, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM
I20260620 19:08:15.882035 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 7107
I20260620 19:08:15.882171 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:08:15.882404 812 external_mini_cluster.cc:1428] Setting key 2b09c066171f4779eb6003fef09527f5
I20260620 19:08:15.882968 6966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:15.874565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:49151 (local address 127.0.203.62:36095)
0620 19:08:15.874653 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:15.874655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:15.874825 (+ 170us) server_negotiation.cc:407] Connection header received
0620 19:08:15.875388 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:15.875389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:15.875401 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:15.875456 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:15.876337 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:15.876833 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:15.877776 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:15.877983 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:15.878041 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:15.880079 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:15.880080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:15.880080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:15.880086 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:15.881549 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:15.881893 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:15.881894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:15.881895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:15.881955 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:15.882288 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:15.882289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:15.882290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:15.882412 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:15.882489 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:15.882796 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 19:08:15.882879 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:08:15.883232 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:15.874528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.2:49151)
0620 19:08:15.874698 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:15.874737 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:15.875266 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:15.875508 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:15.875511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:15.876060 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:15.876255 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:15.876259 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:15.876911 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:15.876912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:15.877686 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:15.877687 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:15.878057 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:15.878057 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:15.878153 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:15.878624 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:15.878631 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:15.879982 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:15.881619 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:15.881623 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:15.881625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:15.881823 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:15.882024 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:15.882024 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:15.882024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:15.882196 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:15.882490 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:15.882497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:15.882648 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 19:08:15.882770 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 19:08:15.882925 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
I20260620 19:08:15.883692 7236 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36095
I20260620 19:08:15.883792 7236 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:15.883975 7236 heartbeater.cc:511] Master 127.0.203.62:36095 requested a full tablet report, sending...
I20260620 19:08:15.884306 6911 ts_manager.cc:194] Registered new tserver with Master: 73d53fd8b46644c4a48293779879f814 (127.0.203.2:42711)
I20260620 19:08:15.884667 6911 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:49151
WARNING: no policy specified for oryx/127.0.203.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.3.keytab.
Entry for principal oryx/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.3.keytab.
I20260620 19:08:15.906767 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36095
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.3.keytab
--principal=oryx/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:15.987317 7241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:15.987514 7241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:15.987535 7241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:15.988597 7241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:15.988631 7241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:15.988662 7241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:08:15.990025 7241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.3.keytab
--principal=oryx/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:15.990206 7241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:15.990374 7241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:15.995255 7246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:15.995255 7249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:15.995282 7247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:15.995560 7241 server_base.cc:1034] running on GCE node
I20260620 19:08:15.995765 7241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:15.995990 7241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:15.997152 7241 hybrid_clock.cc:648] HybridClock initialized: now 1781982495997131 us; error 30 us; skew 500 ppm
Jun 20 19:08:15 dist-test-slave-wd30 krb5kdc[6866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982495, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:15.999658 7241 init.cc:377] Logged in from keytab as oryx/127.0.203.3@KRBTEST.COM (short username oryx)
I20260620 19:08:16.000108 7241 webserver.cc:460] Webserver started at http://127.0.203.3:37665/ using document root <none> and password file <none>
I20260620 19:08:16.000298 7241 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:16.000339 7241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:16.000422 7241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:16.001432 7241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "ec2d760a63644e13bde30b79f7913d16"
format_stamp: "Formatted at 2026-06-20 19:08:15 on dist-test-slave-wd30"
server_key: "20f3b0908a543f7a9bb49b72c399ac2d"
server_key_iv: "dce982afe249801e53e634b11c0f9910"
server_key_version: "encryptionkey@0"
I20260620 19:08:16.001744 7241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "ec2d760a63644e13bde30b79f7913d16"
format_stamp: "Formatted at 2026-06-20 19:08:15 on dist-test-slave-wd30"
server_key: "20f3b0908a543f7a9bb49b72c399ac2d"
server_key_iv: "dce982afe249801e53e634b11c0f9910"
server_key_version: "encryptionkey@0"
I20260620 19:08:16.002982 7241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:08:16.003752 7256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.003957 7241 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 19:08:16.004065 7241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "ec2d760a63644e13bde30b79f7913d16"
format_stamp: "Formatted at 2026-06-20 19:08:15 on dist-test-slave-wd30"
server_key: "20f3b0908a543f7a9bb49b72c399ac2d"
server_key_iv: "dce982afe249801e53e634b11c0f9910"
server_key_version: "encryptionkey@0"
I20260620 19:08:16.004138 7241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:16.032308 7241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:16.034403 7241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:16.034569 7241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:16.034775 7241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:16.035084 7241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:16.035198 7241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.035251 7241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:16.035280 7241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.040995 7241 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:41697
I20260620 19:08:16.041103 7369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:41697 every 8 connection(s)
I20260620 19:08:16.041395 7241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:08:16.041764 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 7241
I20260620 19:08:16.041863 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:08:16.042081 812 external_mini_cluster.cc:1428] Setting key 0ad99abaa07e1550b19eb158e9b38607
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982495, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM
I20260620 19:08:16.050201 6966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.042152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:41195 (local address 127.0.203.62:36095)
0620 19:08:16.042193 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.042195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.042476 (+ 281us) server_negotiation.cc:407] Connection header received
0620 19:08:16.043075 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.043075 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.043085 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.043192 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.044074 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.044572 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.045509 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.045694 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.045752 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.047832 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.047833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.047833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.047839 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.049067 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.049408 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.049409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.049409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.049459 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.049659 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.049659 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.049660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.049762 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.049812 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.050061 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.050130 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:08:16.050309 7372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.042136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.3:41195)
0620 19:08:16.042328 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.042374 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.042984 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.043249 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.043252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.043806 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.043989 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.043993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.044646 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.044646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.045434 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.045436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.045773 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.045774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.045867 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.046335 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.046342 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.047748 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.049135 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.049141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.049142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.049344 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.049516 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.049517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.049517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.049605 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.049815 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.049819 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.049926 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.050006 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.050117 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260620 19:08:16.050766 7370 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36095
I20260620 19:08:16.050877 7370 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:16.051048 7370 heartbeater.cc:511] Master 127.0.203.62:36095 requested a full tablet report, sending...
I20260620 19:08:16.051412 6911 ts_manager.cc:194] Registered new tserver with Master: ec2d760a63644e13bde30b79f7913d16 (127.0.203.3:41697)
I20260620 19:08:16.051740 6911 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:41195
I20260620 19:08:16.053632 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.203.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.61.keytab.
Entry for principal oryx/127.0.203.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.61.keytab.
I20260620 19:08:16.077754 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.61:41813
--webserver_interface=127.0.203.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.61.keytab
--principal=oryx/127.0.203.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:36095,127.0.203.61:41813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:16.158944 7375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:16.159166 7375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:16.159190 7375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:16.160277 7375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:16.160310 7375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:16.160324 7375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:16.160337 7375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:16.160349 7375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:08:16.161811 7375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:36095,127.0.203.61:41813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.61.keytab
--principal=oryx/127.0.203.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.61:41813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.203.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
W20260620 19:08:16.161932 7375 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.203.62:36095,127.0.203.61:41813'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 19:08:16.162040 7375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:16.162207 7375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:16.168992 7381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:16.169013 7383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:16.169396 7380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:16.169416 7375 server_base.cc:1034] running on GCE node
I20260620 19:08:16.169703 7375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:16.170050 7375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:16.171224 7375 hybrid_clock.cc:648] HybridClock initialized: now 1781982496171189 us; error 46 us; skew 500 ppm
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:16.174232 7375 init.cc:377] Logged in from keytab as oryx/127.0.203.61@KRBTEST.COM (short username oryx)
I20260620 19:08:16.174708 7375 webserver.cc:460] Webserver started at http://127.0.203.61:43557/ using document root <none> and password file <none>
I20260620 19:08:16.174948 7375 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:16.175014 7375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:16.175164 7375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:16.176219 7375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/data/instance:
uuid: "324f3d9c6e264f09bd8ad5313baee734"
format_stamp: "Formatted at 2026-06-20 19:08:16 on dist-test-slave-wd30"
server_key: "edf9ff959e8853d812dab6869e4659da"
server_key_iv: "04be10354b0b59c378388348316cc7f8"
server_key_version: "encryptionkey@0"
I20260620 19:08:16.176590 7375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/wal/instance:
uuid: "324f3d9c6e264f09bd8ad5313baee734"
format_stamp: "Formatted at 2026-06-20 19:08:16 on dist-test-slave-wd30"
server_key: "edf9ff959e8853d812dab6869e4659da"
server_key_iv: "04be10354b0b59c378388348316cc7f8"
server_key_version: "encryptionkey@0"
I20260620 19:08:16.177896 7375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:08:16.178588 7390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.178761 7375 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:16.178857 7375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/wal
uuid: "324f3d9c6e264f09bd8ad5313baee734"
format_stamp: "Formatted at 2026-06-20 19:08:16 on dist-test-slave-wd30"
server_key: "edf9ff959e8853d812dab6869e4659da"
server_key_iv: "04be10354b0b59c378388348316cc7f8"
server_key_version: "encryptionkey@0"
I20260620 19:08:16.178943 7375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:16.188346 7375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:16.190601 7375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:16.190764 7375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:16.194615 7375 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.61:41813
I20260620 19:08:16.194657 7442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.61:41813 every 8 connection(s)
I20260620 19:08:16.194999 7375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/data/info.pb
I20260620 19:08:16.196249 7443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:16.198134 7443 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:08:16.202432 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 7375
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.61@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM
I20260620 19:08:16.207372 6966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.198852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44242 (local address 127.0.203.62:36095)
0620 19:08:16.198900 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.198902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.199092 (+ 190us) server_negotiation.cc:407] Connection header received
0620 19:08:16.199609 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.199611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.199630 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.199699 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.200288 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.200780 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.201722 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.201901 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.201953 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.204237 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.204239 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.204240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.204248 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.205653 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.206173 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.206174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.206175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.206245 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.206569 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.206570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.206571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.206812 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.206895 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.207142 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.207228 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:08:16.207480 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.198802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.0.1:44242)
0620 19:08:16.198960 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.199000 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.199506 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.199778 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.199781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.200013 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.200207 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.200211 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.200854 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.200855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.201648 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.201650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.201971 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.201971 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.202073 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.202531 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.202543 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.204137 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.205764 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.205770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.205772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.206040 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.206321 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.206322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.206323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.206498 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.206886 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.206890 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.206997 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.207081 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.207272 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.203.61@KRBTEST.COM
I20260620 19:08:16.208122 7443 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:08:16.211661 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.202937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51606 (local address 127.0.203.61:41813)
0620 19:08:16.203079 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.203083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.203168 (+ 85us) server_negotiation.cc:407] Connection header received
0620 19:08:16.203231 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.203233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.203249 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.203330 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.203954 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.204608 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.205338 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.205670 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.205757 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.208586 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.208593 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.208593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.208599 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.210330 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.210694 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.210696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.210696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.210748 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.210948 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.210949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.210949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.211127 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.211310 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.211439 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.211543 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":88,"threads_started":1}
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.61@KRBTEST.COM for oryx/127.0.203.61@KRBTEST.COM
I20260620 19:08:16.215351 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.208373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:41813 (local address 127.0.0.1:51608)
0620 19:08:16.208412 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.208421 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.208454 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.208700 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.208701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.208893 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.208947 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.208948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.209760 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.209761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.210515 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.210517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.210996 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.210997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.211068 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.211465 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.211470 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.212786 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.214500 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.214501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.214501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.214655 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.214825 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.214825 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.214825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.214872 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.215059 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.215059 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.215089 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.215203 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.215293 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 19:08:16.215463 7448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.208433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51608 (local address 127.0.203.61:41813)
0620 19:08:16.208571 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.208573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.208580 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:08:16.208598 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.208599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.208614 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.208670 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.209026 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.209683 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.210590 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.210929 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.210988 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.212855 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.212856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.212856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.212861 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.214435 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.214721 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.214722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.214722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.214768 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.214923 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.214923 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.214923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.215007 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.215054 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.215229 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.215339 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":119,"thread_start_us":97,"threads_started":1}
I20260620 19:08:16.216704 7443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734: Bootstrap starting.
I20260620 19:08:16.217267 7443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:16.217482 7443 log.cc:824] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:16.218092 7443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734: No bootstrap required, opened a new log
I20260620 19:08:16.219542 7443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } }
I20260620 19:08:16.219666 7443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:16.219689 7443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324f3d9c6e264f09bd8ad5313baee734, State: Initialized, Role: FOLLOWER
I20260620 19:08:16.219810 7443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } }
I20260620 19:08:16.220115 7443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:16.220098 7450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } } }
I20260620 19:08:16.220331 7450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 19:08:16.221968 7461 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:08:16.222047 7461 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:08:16.223711 7375 master_runner.cc:428] Detected that this master 324f3d9c6e264f09bd8ad5313baee734 is joining an existing cluster
I20260620 19:08:16.223771 7375 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.203.61:41813
I20260620 19:08:16.231992 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.224967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:41813 (local address 127.0.0.1:51618)
0620 19:08:16.225148 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.225160 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.225231 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.225380 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.225381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.225603 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.225702 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.225705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.226495 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.226496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.227432 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.227435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.227831 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.227832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.227914 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.228442 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.228448 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.228926 (+ 478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.230814 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.230815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.230815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.231034 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.231286 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.231287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.231287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.231346 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.231634 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.231635 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.231685 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.231766 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.231872 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":106,"threads_started":1}
I20260620 19:08:16.232371 7448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.224995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.0.203.61:41813)
0620 19:08:16.225060 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.225063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.225256 (+ 193us) server_negotiation.cc:407] Connection header received
0620 19:08:16.225291 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.225291 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.225301 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.225357 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.225841 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.226407 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.227519 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.227733 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.227799 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.228997 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.228999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.228999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.229005 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.230728 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.231140 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.231142 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.231142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.231203 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.231418 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.231419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.231419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.231574 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.231643 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.231794 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.231971 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":41}
I20260620 19:08:16.235348 6966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.228648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44256 (local address 127.0.203.62:36095)
0620 19:08:16.228703 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.228706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.228882 (+ 176us) server_negotiation.cc:407] Connection header received
0620 19:08:16.228972 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.228972 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.228985 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.229035 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.229514 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.230156 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.230919 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.231234 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.231295 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.232704 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.232705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.232706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.232714 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.234232 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.234545 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.234546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.234546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.234609 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.234782 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.234783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.234783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.234914 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.234980 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.235172 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.235253 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 19:08:16.235358 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.228586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.0.1:44256)
0620 19:08:16.228793 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.228802 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.228905 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.229047 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.229048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.229314 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.229371 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.229373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.230243 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.230244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.230833 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.230835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.231352 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.231353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.231430 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.232199 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.232204 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.232587 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.234313 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.234314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.234314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.234471 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.234674 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.234675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.234675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.234741 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.234989 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.234990 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.235035 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.235087 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.235212 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":122,"threads_started":1}
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982495, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.62@KRBTEST.COM for oryx/127.0.203.61@KRBTEST.COM
I20260620 19:08:16.246449 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.238460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:41813 (local address 127.0.0.1:51622)
0620 19:08:16.239231 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.239283 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.239359 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.239493 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.239496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.239839 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.239899 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.239901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.240497 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.240498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.241448 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.241451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.241791 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.241792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.241889 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.242311 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.242316 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.243856 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.245440 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.245446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.245448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.245642 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.245846 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.245847 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.245847 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.245898 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.246154 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.246160 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.246202 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.246273 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.246366 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":83,"threads_started":1}
I20260620 19:08:16.246491 7448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.238512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51622 (local address 127.0.203.61:41813)
0620 19:08:16.238548 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.238549 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.239347 (+ 798us) server_negotiation.cc:407] Connection header received
0620 19:08:16.239433 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.239433 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.239442 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.239487 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.239970 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.240421 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.241535 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.241715 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.241771 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.243924 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.243925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.243926 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.243933 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.245351 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.245717 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.245718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.245719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.245778 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.245958 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.245959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.245959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.246090 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.246161 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.246317 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.246397 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:08:16.247258 6911 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.203.61:41813: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734"
member_type: NON_VOTER
last_known_addr {
host: "127.0.203.61"
port: 41813
}
attrs {
promote: true
}
}
dest_uuid: "b58adcd02d204af8810c43fa357d0ffa"
cas_config_opid_index: -1
I20260620 19:08:16.247583 6911 consensus_queue.cc:237] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: NON_VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: true } }
I20260620 19:08:16.248059 7471 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 324f3d9c6e264f09bd8ad5313baee734 (127.0.203.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: NON_VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: true } } }
I20260620 19:08:16.248436 7417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:16.249060 7471 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.203.61:41813
I20260620 19:08:16.249248 7417 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 1 FOLLOWER]: Refusing update from remote peer b58adcd02d204af8810c43fa357d0ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 19:08:16.249259 7470 sys_catalog.cc:455] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b58adcd02d204af8810c43fa357d0ffa" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: NON_VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: true } } }
I20260620 19:08:16.249300 7375 master.cc:537] Master@127.0.203.61:41813 shutting down...
I20260620 19:08:16.249334 7470 sys_catalog.cc:458] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:16.249591 7470 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: NON_VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:08:16.250443 6896 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 19:08:16.250887 6896 consensus_peers.cc:487] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa -> Peer 324f3d9c6e264f09bd8ad5313baee734 (127.0.203.61:41813): Couldn't send request to peer 324f3d9c6e264f09bd8ad5313baee734. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:08:16.252396 7375 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:16.252513 7375 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:16.252569 7375 tablet_replica.cc:330] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734: stopping tablet replica
I20260620 19:08:16.265200 7375 master.cc:552] Master@127.0.203.61:41813 shutdown complete.
I20260620 19:08:16.265277 7375 master_runner.cc:305] Clearing existing system tablet
I20260620 19:08:16.265405 7375 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:16.265468 7375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:16.266203 7375 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:16.266707 7475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.266883 7375 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:16.266953 7375 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.266983 7375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/wal
uuid: "324f3d9c6e264f09bd8ad5313baee734"
format_stamp: "Formatted at 2026-06-20 19:08:16 on dist-test-slave-wd30"
server_key: "edf9ff959e8853d812dab6869e4659da"
server_key_iv: "04be10354b0b59c378388348316cc7f8"
server_key_version: "encryptionkey@0"
I20260620 19:08:16.268118 7375 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:16.269474 7375 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 19:08:16.269556 7375 log.cc:1196] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 19:08:16.269739 7375 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734: Deleting consensus metadata
I20260620 19:08:16.269871 7375 master_runner.cc:315] Copying system tablet from 127.0.203.62:36095
I20260620 19:08:16.270493 7375 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.62:36095
I20260620 19:08:16.276798 6966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.270889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44258 (local address 127.0.203.62:36095)
0620 19:08:16.270941 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.270943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.270983 (+ 40us) server_negotiation.cc:407] Connection header received
0620 19:08:16.271063 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.271064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.271072 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.271154 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.271498 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.271987 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.273049 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.273252 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.273309 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.274335 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.274336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.274336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.274343 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.275717 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.276105 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.276106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.276106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.276154 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.276367 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.276368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.276368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.276466 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.276516 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.276668 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.276733 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 19:08:16.276861 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.270793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.0.1:44258)
0620 19:08:16.270915 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.270928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.270999 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.271151 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.271151 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.271358 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.271426 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.271428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.272066 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.272067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.272954 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.272957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.273330 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.273331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.273401 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.273850 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.273856 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.274248 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.275800 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.275801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.275801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.276027 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.276215 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.276216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.276216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.276304 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.276529 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.276530 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.276575 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.276646 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.276750 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":80,"threads_started":1}
I20260620 19:08:16.277474 6931 tablet_copy_service.cc:140] P b58adcd02d204af8810c43fa357d0ffa: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 324f3d9c6e264f09bd8ad5313baee734 ({username='oryx', principal='oryx/127.0.203.61@KRBTEST.COM'} at 127.0.0.1:44258)
I20260620 19:08:16.277581 6931 tablet_copy_service.cc:161] P b58adcd02d204af8810c43fa357d0ffa: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 324f3d9c6e264f09bd8ad5313baee734 at {username='oryx', principal='oryx/127.0.203.61@KRBTEST.COM'} at 127.0.0.1:44258: session id = 324f3d9c6e264f09bd8ad5313baee734-00000000000000000000000000000000
I20260620 19:08:16.278429 6931 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:08:16.279184 7375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 19:08:16.280148 7375 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734: tablet copy: Starting download of 0 data blocks...
I20260620 19:08:16.280344 7375 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734: tablet copy: Starting download of 1 WAL segments...
I20260620 19:08:16.281276 7375 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:08:16.282081 6931 tablet_copy_service.cc:342] P b58adcd02d204af8810c43fa357d0ffa: Request end of tablet copy session 324f3d9c6e264f09bd8ad5313baee734-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.203.61@KRBTEST.COM'} at 127.0.0.1:44258
I20260620 19:08:16.282162 6931 tablet_copy_service.cc:433] P b58adcd02d204af8810c43fa357d0ffa: ending tablet copy session 324f3d9c6e264f09bd8ad5313baee734-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 324f3d9c6e264f09bd8ad5313baee734
I20260620 19:08:16.284569 7375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:16.285782 7485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:16.285856 7483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:16.286018 7375 server_base.cc:1034] running on GCE node
W20260620 19:08:16.286046 7482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:16.286192 7375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:16.286351 7375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:16.287482 7375 hybrid_clock.cc:648] HybridClock initialized: now 1781982496287471 us; error 23 us; skew 500 ppm
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:16.288993 7375 init.cc:377] Logged in from keytab as oryx/127.0.203.61@KRBTEST.COM (short username oryx)
I20260620 19:08:16.289305 7375 webserver.cc:460] Webserver started at http://127.0.203.61:41059/ using document root <none> and password file <none>
I20260620 19:08:16.289425 7375 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:16.289482 7375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:16.290117 7375 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:08:16.290642 7492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.290764 7375 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:08:16.290812 7375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/wal
uuid: "324f3d9c6e264f09bd8ad5313baee734"
format_stamp: "Formatted at 2026-06-20 19:08:16 on dist-test-slave-wd30"
server_key: "edf9ff959e8853d812dab6869e4659da"
server_key_iv: "04be10354b0b59c378388348316cc7f8"
server_key_version: "encryptionkey@0"
I20260620 19:08:16.290868 7375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:16.308399 7375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:16.310039 7375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:16.313241 7375 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.61:41813
I20260620 19:08:16.313299 7544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.61:41813 every 8 connection(s)
I20260620 19:08:16.313781 7375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-1/data/info.pb
I20260620 19:08:16.314553 7546 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:08:16.315068 7546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734: Bootstrap starting.
I20260620 19:08:16.317687 7546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:08:16.317956 7546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734: Bootstrap complete.
I20260620 19:08:16.318197 7546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: NON_VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: true } }
I20260620 19:08:16.318293 7546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 324f3d9c6e264f09bd8ad5313baee734, State: Initialized, Role: LEARNER
I20260620 19:08:16.318338 7546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: NON_VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: true } }
I20260620 19:08:16.318518 7546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:16.320624 7560 catalog_manager.cc:1259] Loaded cluster ID: b794508261304e84b9d0314b8f040e46
I20260620 19:08:16.320688 7560 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734: loading cluster ID for follower catalog manager: success
I20260620 19:08:16.321831 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.313378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51648 (local address 127.0.203.61:41813)
0620 19:08:16.313515 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.313517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.313527 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:08:16.313554 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.313555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.313566 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.313639 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.314366 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.315355 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.316376 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.316684 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.316763 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.318208 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.318209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.318209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.318217 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.320165 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.320616 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.320617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.320617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.320723 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.321024 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.321025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.321025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.321157 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.321251 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.321635 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.321715 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260620 19:08:16.323782 7560 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734: acquiring CA information for follower catalog manager: success
I20260620 19:08:16.324110 7560 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:08:16.327286 7549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: NON_VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: true } } }
I20260620 19:08:16.327399 7549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [sys.catalog]: This master's current role is: LEARNER
WARNING: no policy specified for oryx/127.0.203.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.60.keytab.
Entry for principal oryx/127.0.203.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.60.keytab.
I20260620 19:08:16.350977 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.60:42511
--webserver_interface=127.0.203.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.60.keytab
--principal=oryx/127.0.203.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:36095,127.0.203.61:41813,127.0.203.60:42511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:16.434744 7563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:16.434947 7563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:16.434969 7563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:16.436270 7563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:16.436316 7563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:16.436329 7563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:16.436342 7563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:16.436383 7563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:08:16.437860 7563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:36095,127.0.203.61:41813,127.0.203.60:42511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.60.keytab
--principal=oryx/127.0.203.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.60:42511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.203.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:16.438122 7563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:16.438386 7563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:16.443091 7569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:16.443097 7568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:16.443365 7563 server_base.cc:1034] running on GCE node
W20260620 19:08:16.443099 7571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:16.443702 7563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:16.444073 7563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:16.445256 7563 hybrid_clock.cc:648] HybridClock initialized: now 1781982496445247 us; error 49 us; skew 500 ppm
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:16.447875 7563 init.cc:377] Logged in from keytab as oryx/127.0.203.60@KRBTEST.COM (short username oryx)
I20260620 19:08:16.448402 7563 webserver.cc:460] Webserver started at http://127.0.203.60:35997/ using document root <none> and password file <none>
I20260620 19:08:16.448647 7563 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:16.448714 7563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:16.448866 7563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:16.449940 7563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/data/instance:
uuid: "d780f2401c23420e90ff49507df4851a"
format_stamp: "Formatted at 2026-06-20 19:08:16 on dist-test-slave-wd30"
server_key: "b4f77a4eaa0a363ccfcf507a2de02e28"
server_key_iv: "4bd76f1f017ee3c586f0b27f2882ca51"
server_key_version: "encryptionkey@0"
I20260620 19:08:16.450304 7563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/wal/instance:
uuid: "d780f2401c23420e90ff49507df4851a"
format_stamp: "Formatted at 2026-06-20 19:08:16 on dist-test-slave-wd30"
server_key: "b4f77a4eaa0a363ccfcf507a2de02e28"
server_key_iv: "4bd76f1f017ee3c586f0b27f2882ca51"
server_key_version: "encryptionkey@0"
I20260620 19:08:16.451743 7563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:08:16.452611 7578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.452883 7563 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:08:16.452992 7563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/wal
uuid: "d780f2401c23420e90ff49507df4851a"
format_stamp: "Formatted at 2026-06-20 19:08:16 on dist-test-slave-wd30"
server_key: "b4f77a4eaa0a363ccfcf507a2de02e28"
server_key_iv: "4bd76f1f017ee3c586f0b27f2882ca51"
server_key_version: "encryptionkey@0"
I20260620 19:08:16.453078 7563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:16.473824 7563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:16.475754 7563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:16.475922 7563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:16.480796 7563 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.60:42511
I20260620 19:08:16.480834 7630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.60:42511 every 8 connection(s)
I20260620 19:08:16.481216 7563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/data/info.pb
I20260620 19:08:16.482177 7631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:16.483773 7631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:08:16.486286 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 7563
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.60@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.203.60@KRBTEST.COM
I20260620 19:08:16.493165 6966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.484629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42416 (local address 127.0.203.62:36095)
0620 19:08:16.484683 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.484685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.484715 (+ 30us) server_negotiation.cc:407] Connection header received
0620 19:08:16.485158 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.485159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.485168 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.485240 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.485808 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.486306 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.487589 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.487858 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.487909 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.490040 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.490042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.490042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.490050 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.491706 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.492189 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.492190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.492190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.492256 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.492518 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.492519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.492519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.492668 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.492751 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.492992 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.493075 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:08:16.493356 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.484446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.0.1:42416)
0620 19:08:16.484618 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.484653 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.485075 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.485298 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.485301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.485541 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.485726 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.485741 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.486433 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.486434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.487500 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.487502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.487932 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.487933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.488064 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.488583 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.488593 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.489959 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.491794 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.491800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.491802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.492120 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.492325 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.492326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.492327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.492453 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.492750 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.492754 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.492859 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.492963 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.493103 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260620 19:08:16.493942 7631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:08:16.495941 7634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.486818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.0.203.60:42511)
0620 19:08:16.486989 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.486993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.487008 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:08:16.487025 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.487027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.487042 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.487127 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.487760 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.488698 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.489402 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.489818 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.489908 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.492821 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.492828 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.492828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.492835 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.494729 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.495082 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.495086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.495086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.495162 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.495356 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.495357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.495357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.495494 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.495652 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.495742 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.495818 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.60@KRBTEST.COM for oryx/127.0.203.61@KRBTEST.COM
I20260620 19:08:16.501037 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.494186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:41813 (local address 127.0.0.1:52054)
0620 19:08:16.494250 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.494260 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.494313 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.494443 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.494443 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.494597 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.494636 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.494637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.495398 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.495399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.495877 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.495878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.496343 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.496344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.496414 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.496760 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.496764 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.497804 (+ 1040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.499835 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.499836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.499836 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.500081 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.500330 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.500331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.500331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.500394 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.500693 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.500694 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.500734 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.500821 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.500925 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 19:08:16.501084 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.494272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52054 (local address 127.0.203.61:41813)
0620 19:08:16.494337 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.494339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.494350 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:08:16.494374 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.494375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.494388 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.494432 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.494712 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.495309 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.495972 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.496267 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.496334 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.497890 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.497892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.497892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.497900 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.499706 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.500177 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.500179 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.500179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.500241 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.500478 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.500479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.500479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.500623 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.500694 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.500867 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.500958 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:08:16.501610 7631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.60@KRBTEST.COM for oryx/127.0.203.60@KRBTEST.COM
I20260620 19:08:16.511718 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.501871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:42511 (local address 127.0.0.1:37622)
0620 19:08:16.501902 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.501913 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.501966 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.502086 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.502086 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.502275 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.502326 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.502328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.503179 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.503180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.504035 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.504037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.504500 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.504500 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.504538 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.504906 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.504910 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.506262 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.510677 (+ 4415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.510678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.510679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.510886 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.511103 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.511104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.511104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.511177 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.511423 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.511424 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.511459 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.511517 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.511608 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 19:08:16.511765 7634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.501901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37622 (local address 127.0.203.60:42511)
0620 19:08:16.501938 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.501940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.501948 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:08:16.502025 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.502026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.502036 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.502082 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.502403 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.503067 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.504123 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.504400 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.504461 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.506341 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.506342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.506343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.506349 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.510603 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.510973 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.510974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.510974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.511031 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.511252 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.511252 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.511253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.511355 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.511432 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.511546 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.511656 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":19}
I20260620 19:08:16.513820 7631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: Bootstrap starting.
I20260620 19:08:16.514621 7631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:16.514904 7631 log.cc:824] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:16.515662 7631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: No bootstrap required, opened a new log
I20260620 19:08:16.517539 7631 raft_consensus.cc:348] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } }
I20260620 19:08:16.517665 7631 raft_consensus.cc:374] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:16.517688 7631 raft_consensus.cc:729] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d780f2401c23420e90ff49507df4851a, State: Initialized, Role: FOLLOWER
I20260620 19:08:16.517865 7631 consensus_queue.cc:260] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } }
I20260620 19:08:16.518141 7636 sys_catalog.cc:455] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } } }
I20260620 19:08:16.518181 7631 sys_catalog.cc:564] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:16.518247 7636 sys_catalog.cc:458] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [sys.catalog]: This master's current role is: FOLLOWER
W20260620 19:08:16.520134 7647 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:08:16.520198 7647 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:08:16.522322 7563 master_runner.cc:428] Detected that this master d780f2401c23420e90ff49507df4851a is joining an existing cluster
I20260620 19:08:16.522362 7563 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.203.60:42511
I20260620 19:08:16.530233 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.523756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52060 (local address 127.0.203.61:41813)
0620 19:08:16.523851 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.523853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.523861 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:08:16.523883 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.523884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.523893 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.523951 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.524454 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.525192 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.526281 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.526571 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.526611 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.527719 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.527721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.527721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.527728 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.529192 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.529569 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.529570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.529570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.529617 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.529803 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.529804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.529804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.529895 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.529945 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.530117 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.530169 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 19:08:16.530918 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.523444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:41813 (local address 127.0.0.1:52060)
0620 19:08:16.523686 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.523695 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.523740 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.523994 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.523995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.524299 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.524361 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.524363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.525302 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.525302 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.526171 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.526173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.526684 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.526685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.526757 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.527161 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.527166 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.527624 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.529281 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.529282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.529282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.529499 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.529681 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.529681 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.529682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.529743 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.529958 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.529959 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.530002 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.530103 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.530197 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":6,"thread_start_us":72,"threads_started":1}
I20260620 19:08:16.547436 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 6973
I20260620 19:08:16.552874 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:40925
--local_ip_for_outbound_sockets=127.0.203.1
--tserver_master_addrs=127.0.203.62:36095
--webserver_port=41453
--webserver_interface=127.0.203.1
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.1.keytab
--principal=oryx/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.203.62:36095,127.0.203.61:41813
--tserver_master_addrs=127.0.203.62:36095,127.0.203.61:41813,127.0.203.60:42511 with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:08:16.555112 7634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.523802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37624 (local address 127.0.203.60:42511)
0620 19:08:16.523837 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.523838 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.523845 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:08:16.523861 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.523862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.523870 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.523915 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.524378 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.525108 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.526274 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.526536 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.526573 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.552414 (+ 25841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.552416 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.552416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.552422 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.553976 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.554340 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.554341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.554342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.554384 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.554548 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.554549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.554549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.554674 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.554712 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.554886 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.554962 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:08:16.555255 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.523604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:42511 (local address 127.0.0.1:37624)
0620 19:08:16.523691 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.523701 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.523758 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.523952 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.523953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.524253 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.524304 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.524306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.525195 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.525196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.526182 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.526185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.526654 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.526655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.526707 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.551780 (+ 25073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.551786 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.552304 (+ 518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.554066 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.554066 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.554067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.554269 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.554443 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.554443 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.554444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.554490 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.554748 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.554748 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.554793 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.555006 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.555089 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":65,"thread_start_us":51,"threads_started":1}
I20260620 19:08:16.559590 6966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.523465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.0.203.62:36095)
0620 19:08:16.523515 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.523518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.523639 (+ 121us) server_negotiation.cc:407] Connection header received
0620 19:08:16.523875 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.523876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.523888 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.523919 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.524388 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.525109 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.526285 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.526582 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.526620 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.552555 (+ 25935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.552557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.552557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.552565 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.554876 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.555587 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.555589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.555590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.555653 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.555873 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.555874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.555874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.556069 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.556136 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.559392 (+ 3256us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.559481 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:08:16.559947 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.523419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.0.1:42430)
0620 19:08:16.523580 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.523590 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.523798 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.523980 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.523980 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.524218 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.524294 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.524295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.525195 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.525196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.526186 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.526187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.526689 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.526690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.526742 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.551897 (+ 25155us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.551904 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.552443 (+ 539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.554984 (+ 2541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.554985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.554985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.555495 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.555731 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.555732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.555733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.555794 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.559196 (+ 3402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.559197 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.559253 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.559696 (+ 443us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.559804 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":69,"thread_start_us":77,"threads_started":1}
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982495, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.62@KRBTEST.COM for oryx/127.0.203.60@KRBTEST.COM
I20260620 19:08:16.570384 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.562481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:42511 (local address 127.0.0.1:37640)
0620 19:08:16.562554 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.562564 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.562598 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.562868 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.562869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.563034 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.563082 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.563084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.564058 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.564059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.564908 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.564911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.565458 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.565459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.565533 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.565943 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.565947 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.567589 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.569325 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.569326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.569327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.569542 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.569744 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.569745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.569745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.569807 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.570060 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.570060 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.570106 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.570173 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.570281 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 19:08:16.570385 7634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.562661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37640 (local address 127.0.203.60:42511)
0620 19:08:16.562725 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.562727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.562737 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:08:16.562771 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.562772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.562784 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.562854 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.563179 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.563963 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.564997 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.565357 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.565457 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.567675 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.567676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.567677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.567684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.569230 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.569617 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.569618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.569618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.569674 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.569870 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.569871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.569872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.569993 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.570060 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.570212 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.570293 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:08:16.571178 6911 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.203.60:42511: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d780f2401c23420e90ff49507df4851a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.203.60"
port: 42511
}
attrs {
promote: true
}
}
dest_uuid: "b58adcd02d204af8810c43fa357d0ffa"
cas_config_opid_index: 5
I20260620 19:08:16.571429 6911 consensus_queue.cc:237] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: NON_VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: true } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: NON_VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: true } }
I20260620 19:08:16.571798 7470 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d780f2401c23420e90ff49507df4851a (127.0.203.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: NON_VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: true } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: NON_VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: true } } }
I20260620 19:08:16.572610 7605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:16.572764 7470 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.203.60:42511
I20260620 19:08:16.572834 7470 sys_catalog.cc:455] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b58adcd02d204af8810c43fa357d0ffa" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: NON_VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: true } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: NON_VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: true } } }
I20260620 19:08:16.572894 7470 sys_catalog.cc:458] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:16.573500 7563 master.cc:537] Master@127.0.203.60:42511 shutting down...
I20260620 19:08:16.573514 7605 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 1 FOLLOWER]: Refusing update from remote peer b58adcd02d204af8810c43fa357d0ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 19:08:16.573874 7471 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: NON_VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:08:16.574487 6896 consensus_peers.cc:487] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa -> Peer d780f2401c23420e90ff49507df4851a (127.0.203.60:42511): Couldn't send request to peer d780f2401c23420e90ff49507df4851a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:08:16.577731 7563 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:16.577824 7563 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:16.577865 7563 tablet_replica.cc:330] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: stopping tablet replica
I20260620 19:08:16.578280 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.571962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52068 (local address 127.0.203.61:41813)
0620 19:08:16.572002 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.572004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.573612 (+ 1608us) server_negotiation.cc:407] Connection header received
0620 19:08:16.574636 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.574638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.574661 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.574699 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:16.574885 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.575657 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.577089 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.577700 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.577773 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.577845 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.578153 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.578195 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:08:16.579016 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.571869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:41813 (local address 127.0.0.1:52068)
0620 19:08:16.573548 (+ 1679us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.573559 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.574553 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.574778 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.574779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.574780 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:16.574818 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.574820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.575765 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.575766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.576991 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.576994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.577973 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.577974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.578044 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.578069 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.578551 (+ 482us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.578627 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1660}
I20260620 19:08:16.579545 7519 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 1 LEARNER]: Refusing update from remote peer b58adcd02d204af8810c43fa357d0ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 19:08:16.579931 7471 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: NON_VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:08:16.580492 7519 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d780f2401c23420e90ff49507df4851a (127.0.203.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: NON_VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: true } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: NON_VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: true } } }
I20260620 19:08:16.581075 7563 master.cc:552] Master@127.0.203.60:42511 shutdown complete.
I20260620 19:08:16.581110 7563 master_runner.cc:305] Clearing existing system tablet
I20260620 19:08:16.581245 7563 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:16.581278 7563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:16.581557 7549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b58adcd02d204af8810c43fa357d0ffa. Latest consensus state: current_term: 1 leader_uuid: "b58adcd02d204af8810c43fa357d0ffa" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: NON_VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: true } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: NON_VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: true } } }
I20260620 19:08:16.581637 7549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [sys.catalog]: This master's current role is: LEARNER
I20260620 19:08:16.581668 7549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b58adcd02d204af8810c43fa357d0ffa" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: NON_VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: true } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: NON_VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: true } } }
I20260620 19:08:16.581707 7549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [sys.catalog]: This master's current role is: LEARNER
I20260620 19:08:16.581890 7563 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:16.582319 7660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.582326 7471 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa: attempting to promote NON_VOTER 324f3d9c6e264f09bd8ad5313baee734 to VOTER
I20260620 19:08:16.582402 7563 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:16.582437 7563 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.582455 7563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/wal
uuid: "d780f2401c23420e90ff49507df4851a"
format_stamp: "Formatted at 2026-06-20 19:08:16 on dist-test-slave-wd30"
server_key: "b4f77a4eaa0a363ccfcf507a2de02e28"
server_key_iv: "4bd76f1f017ee3c586f0b27f2882ca51"
server_key_version: "encryptionkey@0"
I20260620 19:08:16.582506 7471 consensus_queue.cc:237] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: NON_VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: true } }
I20260620 19:08:16.583091 7563 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:16.583379 7519 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 1 LEARNER]: Refusing update from remote peer b58adcd02d204af8810c43fa357d0ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 19:08:16.583604 7470 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:08:16.584013 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.583226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:42511 (local address 127.0.0.1:37654)
0620 19:08:16.583942 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.583986 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":699}
I20260620 19:08:16.584362 7563 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 19:08:16.584369 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.584297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:42511 (local address 127.0.0.1:37660)
0620 19:08:16.584327 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.584348 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 19:08:16.584422 7563 log.cc:1196] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
W20260620 19:08:16.584515 6896 consensus_peers.cc:487] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa -> Peer d780f2401c23420e90ff49507df4851a (127.0.203.60:42511): Couldn't send request to peer d780f2401c23420e90ff49507df4851a. Status: Network error: Client connection negotiation failed: client connection to 127.0.203.60:42511: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 19:08:16.584592 7563 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: Deleting consensus metadata
I20260620 19:08:16.584708 7563 master_runner.cc:315] Copying system tablet from 127.0.203.62:36095
I20260620 19:08:16.584946 7470 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 324f3d9c6e264f09bd8ad5313baee734 (127.0.203.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: NON_VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: true } } }
I20260620 19:08:16.585276 7563 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.62:36095
I20260620 19:08:16.585749 7470 sys_catalog.cc:455] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b58adcd02d204af8810c43fa357d0ffa" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: NON_VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: true } } }
I20260620 19:08:16.585816 7470 sys_catalog.cc:458] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:16.585861 7470 sys_catalog.cc:455] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b58adcd02d204af8810c43fa357d0ffa" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: NON_VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: true } } }
I20260620 19:08:16.585899 7470 sys_catalog.cc:458] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:16.586284 7519 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 324f3d9c6e264f09bd8ad5313baee734 (127.0.203.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: NON_VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: true } } }
I20260620 19:08:16.587271 7658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b58adcd02d204af8810c43fa357d0ffa" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: NON_VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: true } } }
I20260620 19:08:16.587416 7658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:08:16.594625 6966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.587787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42438 (local address 127.0.203.62:36095)
0620 19:08:16.587826 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.587829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.587959 (+ 130us) server_negotiation.cc:407] Connection header received
0620 19:08:16.587992 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.587993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.588005 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.588065 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.588447 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.589217 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.590226 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.590555 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.590621 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.591755 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.591756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.591757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.591764 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.593434 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.593816 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.593817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.593817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.593879 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.594085 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.594086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.594086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.594218 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.594299 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.594459 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.594538 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:08:16.594664 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.587732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.0.1:42438)
0620 19:08:16.587870 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.587881 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.587948 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.588085 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.588086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.588317 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.588371 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.588374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.589311 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.589312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.590109 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.590111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.590630 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.590631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.590703 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.591261 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.591266 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.591657 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.593515 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.593516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.593517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.593736 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.593951 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.593952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.593952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.594017 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.594300 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.594301 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.594347 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.594420 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.594570 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":51,"threads_started":1}
I20260620 19:08:16.595191 6931 tablet_copy_service.cc:140] P b58adcd02d204af8810c43fa357d0ffa: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d780f2401c23420e90ff49507df4851a ({username='oryx', principal='oryx/127.0.203.60@KRBTEST.COM'} at 127.0.0.1:42438)
I20260620 19:08:16.595254 6931 tablet_copy_service.cc:161] P b58adcd02d204af8810c43fa357d0ffa: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d780f2401c23420e90ff49507df4851a at {username='oryx', principal='oryx/127.0.203.60@KRBTEST.COM'} at 127.0.0.1:42438: session id = d780f2401c23420e90ff49507df4851a-00000000000000000000000000000000
I20260620 19:08:16.595798 6931 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:08:16.596297 7563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 19:08:16.597532 7563 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: tablet copy: Starting download of 0 data blocks...
I20260620 19:08:16.597656 7563 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: tablet copy: Starting download of 1 WAL segments...
I20260620 19:08:16.598512 7563 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:08:16.599428 6931 tablet_copy_service.cc:342] P b58adcd02d204af8810c43fa357d0ffa: Request end of tablet copy session d780f2401c23420e90ff49507df4851a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.203.60@KRBTEST.COM'} at 127.0.0.1:42438
I20260620 19:08:16.599485 6931 tablet_copy_service.cc:433] P b58adcd02d204af8810c43fa357d0ffa: ending tablet copy session d780f2401c23420e90ff49507df4851a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d780f2401c23420e90ff49507df4851a
I20260620 19:08:16.601794 7563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:16.603099 7668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:16.607604 7669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:16.607913 7671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:16.607923 7563 server_base.cc:1034] running on GCE node
I20260620 19:08:16.608150 7563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:16.608289 7563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:16.609409 7563 hybrid_clock.cc:648] HybridClock initialized: now 1781982496609423 us; error 50 us; skew 500 ppm
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:16.610908 7563 init.cc:377] Logged in from keytab as oryx/127.0.203.60@KRBTEST.COM (short username oryx)
I20260620 19:08:16.611184 7563 webserver.cc:460] Webserver started at http://127.0.203.60:35239/ using document root <none> and password file <none>
I20260620 19:08:16.611310 7563 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:16.611351 7563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:16.612097 7563 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:08:16.612682 7678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.612854 7563 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:08:16.612906 7563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/wal
uuid: "d780f2401c23420e90ff49507df4851a"
format_stamp: "Formatted at 2026-06-20 19:08:16 on dist-test-slave-wd30"
server_key: "b4f77a4eaa0a363ccfcf507a2de02e28"
server_key_iv: "4bd76f1f017ee3c586f0b27f2882ca51"
server_key_version: "encryptionkey@0"
I20260620 19:08:16.612970 7563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:16.641203 7563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:16.643093 7563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:16.646323 7563 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.60:42511
I20260620 19:08:16.646425 7730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.60:42511 every 8 connection(s)
I20260620 19:08:16.646679 7563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/master-2/data/info.pb
I20260620 19:08:16.647679 7731 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:08:16.648226 7731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: Bootstrap starting.
I20260620 19:08:16.651214 7731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:08:16.651423 7731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: Bootstrap complete.
I20260620 19:08:16.651618 7731 raft_consensus.cc:348] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: NON_VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: true } }
I20260620 19:08:16.651707 7731 raft_consensus.cc:729] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d780f2401c23420e90ff49507df4851a, State: Initialized, Role: LEARNER
I20260620 19:08:16.651746 7731 consensus_queue.cc:260] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: NON_VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: true } }
I20260620 19:08:16.651926 7731 sys_catalog.cc:564] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:16.651917 7734 sys_catalog.cc:455] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: NON_VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: true } } }
I20260620 19:08:16.652151 7734 sys_catalog.cc:458] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [sys.catalog]: This master's current role is: LEARNER
I20260620 19:08:16.653510 7745 catalog_manager.cc:1259] Loaded cluster ID: b794508261304e84b9d0314b8f040e46
I20260620 19:08:16.653600 7745 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: loading cluster ID for follower catalog manager: success
I20260620 19:08:16.654928 7745 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: acquiring CA information for follower catalog manager: success
I20260620 19:08:16.655253 7745 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 19:08:16.661612 7656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:16.661798 7656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:16.661819 7656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:16.663015 7656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:16.663056 7656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:16.663089 7656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:08:16.664686 7656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.1.keytab
--principal=oryx/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:40925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=41453
--tserver_master_addrs=127.0.203.62:36095,127.0.203.61:41813,127.0.203.60:42511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:16.664908 7656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:16.665154 7656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:16.671689 7751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:16.671666 7750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:16.671871 7656 server_base.cc:1034] running on GCE node
W20260620 19:08:16.671693 7753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:16.672224 7656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:16.672497 7656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:16.673667 7656 hybrid_clock.cc:648] HybridClock initialized: now 1781982496673655 us; error 28 us; skew 500 ppm
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:16.676287 7656 init.cc:377] Logged in from keytab as oryx/127.0.203.1@KRBTEST.COM (short username oryx)
I20260620 19:08:16.676788 7656 webserver.cc:460] Webserver started at http://127.0.203.1:41453/ using document root <none> and password file <none>
I20260620 19:08:16.677027 7656 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:16.677107 7656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:16.678444 7656 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:16.679029 7760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.679236 7656 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:16.679333 7656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "6a241202a70f420ca87abfce500521fa"
format_stamp: "Formatted at 2026-06-20 19:08:15 on dist-test-slave-wd30"
server_key: "1ce971251b0292d83ebf101c3e097d02"
server_key_iv: "30e4ce31c71a9d1b6a524de80df26589"
server_key_version: "encryptionkey@0"
I20260620 19:08:16.679684 7656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:16.697579 7656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:16.699420 7656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:16.699575 7656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:16.699805 7656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:16.700109 7656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:16.700162 7656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.700212 7656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:16.700238 7656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.706491 7656 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:40925
I20260620 19:08:16.706578 7873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:40925 every 8 connection(s)
I20260620 19:08:16.706952 7656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:08:16.711169 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 7656
I20260620 19:08:16.711325 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 7107
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.60@KRBTEST.COM
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM
I20260620 19:08:16.717767 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:42711
--local_ip_for_outbound_sockets=127.0.203.2
--tserver_master_addrs=127.0.203.62:36095
--webserver_port=39119
--webserver_interface=127.0.203.2
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.2.keytab
--principal=oryx/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.203.62:36095,127.0.203.61:41813
--tserver_master_addrs=127.0.203.62:36095,127.0.203.61:41813,127.0.203.60:42511 with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:08:16.719595 6966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.708171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:41389 (local address 127.0.203.62:36095)
0620 19:08:16.708229 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.708231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.708240 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:08:16.709052 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.709054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.709074 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.709164 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.710204 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.710709 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.712095 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.712395 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.712439 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.715865 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.715866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.715866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.715874 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.717847 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.718339 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.718340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.718341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.718408 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.718734 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.718735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.718736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.718914 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.718960 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.719397 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.719491 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 19:08:16.719810 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.707717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.1:41389)
0620 19:08:16.707870 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.707911 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.708927 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.709286 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.709289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.709713 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.710027 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.710032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.710818 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.710818 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.711974 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.711976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.712589 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.712590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.712761 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.713555 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.713559 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.715764 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.717923 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.717929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.717931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.718247 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.718490 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.718491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.718491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.718632 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.719031 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.719039 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.719239 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.719354 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.719572 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":49,"thread_start_us":84,"threads_started":1}
I20260620 19:08:16.720096 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.708106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:50723 (local address 127.0.203.60:42511)
0620 19:08:16.708209 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.708211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.709158 (+ 947us) server_negotiation.cc:407] Connection header received
0620 19:08:16.709237 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.709238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.709254 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.709330 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.710133 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.711176 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.712185 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.712499 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.712560 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.715790 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.715791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.715792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.715800 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.718570 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.718985 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.718986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.718986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.719055 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.719302 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.719303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.719303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.719442 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.719513 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.719897 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.719986 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":46,"threads_started":1}
I20260620 19:08:16.720213 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.708004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:42511 (local address 127.0.203.1:50723)
0620 19:08:16.709103 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.709112 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.709162 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.709341 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.709342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.709731 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.710027 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.710032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.711267 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.711268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.712096 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.712097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.712613 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.712614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.712761 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.713438 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.713449 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.715673 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.718653 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.718654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.718655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.718873 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.719160 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.719161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.719161 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.719230 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.719542 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.719543 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.719588 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.719669 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.719801 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1082,"mutex_wait_us":3,"thread_start_us":78,"threads_started":1}
I20260620 19:08:16.720315 7875 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36095
I20260620 19:08:16.720398 7875 heartbeater.cc:464] Registering TS with master...
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.61@KRBTEST.COM
I20260620 19:08:16.720573 7875 heartbeater.cc:511] Master 127.0.203.62:36095 requested a full tablet report, sending...
I20260620 19:08:16.721057 6911 ts_manager.cc:194] Re-registered known tserver with Master: 6a241202a70f420ca87abfce500521fa (127.0.203.1:40925)
I20260620 19:08:16.721578 6911 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:41389
I20260620 19:08:16.721832 7874 heartbeater.cc:347] Connected to a master server at 127.0.203.60:42511
I20260620 19:08:16.721889 7874 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:16.721956 7874 heartbeater.cc:511] Master 127.0.203.60:42511 requested a full tablet report, sending...
I20260620 19:08:16.722352 7695 ts_manager.cc:194] Registered new tserver with Master: 6a241202a70f420ca87abfce500521fa (127.0.203.1:40925)
I20260620 19:08:16.724025 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.707781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:34031 (local address 127.0.203.61:41813)
0620 19:08:16.707864 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.707866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.708221 (+ 355us) server_negotiation.cc:407] Connection header received
0620 19:08:16.709076 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.709077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.709091 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.709150 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.710137 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.710916 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.716766 (+ 5850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.717111 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.717153 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.720969 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.720970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.720971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.720979 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.722718 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.723115 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.723138 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.723139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.723210 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.723463 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.723464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.723465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.723594 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.723677 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.723843 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.723929 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:08:16.724267 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.707732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:41813 (local address 127.0.203.1:34031)
0620 19:08:16.708145 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.708157 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.708948 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.709284 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.709289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.709724 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.710027 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.710032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.715771 (+ 5739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.715771 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.716674 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.716677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.717237 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.717238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.717311 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.719370 (+ 2059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.719375 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.720879 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.722804 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.722805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.722805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.723027 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.723307 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.723307 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.723308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.723380 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.723667 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.723668 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.723710 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.723816 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.723929 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":103,"thread_start_us":122,"threads_started":1}
I20260620 19:08:16.724624 7876 heartbeater.cc:347] Connected to a master server at 127.0.203.61:41813
I20260620 19:08:16.724684 7876 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:16.724736 7876 heartbeater.cc:511] Master 127.0.203.61:41813 requested a full tablet report, sending...
I20260620 19:08:16.725087 7509 ts_manager.cc:194] Registered new tserver with Master: 6a241202a70f420ca87abfce500521fa (127.0.203.1:40925)
W20260620 19:08:16.801318 7882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:16.801534 7882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:16.801573 7882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:16.802726 7882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:16.802791 7882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:16.802835 7882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:08:16.804463 7882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.2.keytab
--principal=oryx/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:42711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=39119
--tserver_master_addrs=127.0.203.62:36095,127.0.203.61:41813,127.0.203.60:42511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:16.804715 7882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:16.804931 7882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:16.810020 7887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:16.810026 7888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:16.810026 7890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:16.810149 7882 server_base.cc:1034] running on GCE node
I20260620 19:08:16.810527 7882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:16.810817 7882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:16.811975 7882 hybrid_clock.cc:648] HybridClock initialized: now 1781982496811950 us; error 37 us; skew 500 ppm
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:16.814563 7882 init.cc:377] Logged in from keytab as oryx/127.0.203.2@KRBTEST.COM (short username oryx)
I20260620 19:08:16.815023 7882 webserver.cc:460] Webserver started at http://127.0.203.2:39119/ using document root <none> and password file <none>
I20260620 19:08:16.815323 7882 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:16.815408 7882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:16.816744 7882 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:16.817361 7897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.817534 7882 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:16.817632 7882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "73d53fd8b46644c4a48293779879f814"
format_stamp: "Formatted at 2026-06-20 19:08:15 on dist-test-slave-wd30"
server_key: "0123ea4c3d356d53c14a29d4dabf0ddf"
server_key_iv: "34bb0be9e3e36090e9dde65d340d917a"
server_key_version: "encryptionkey@0"
I20260620 19:08:16.817965 7882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:16.840044 7882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:16.842386 7882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:16.842550 7882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:16.842796 7882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:16.843106 7882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:16.843211 7882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.843263 7882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:16.843310 7882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.849570 7882 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:42711
I20260620 19:08:16.849615 8010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:42711 every 8 connection(s)
I20260620 19:08:16.849944 7882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:08:16.855763 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 7882
I20260620 19:08:16.855952 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 7241
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.61@KRBTEST.COM
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.60@KRBTEST.COM
I20260620 19:08:16.861900 6966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.850639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:44997 (local address 127.0.203.62:36095)
0620 19:08:16.850692 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.850695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.850957 (+ 262us) server_negotiation.cc:407] Connection header received
0620 19:08:16.851567 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.851569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.851582 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.851632 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.852557 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.853315 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.854480 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.854781 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.854841 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.858444 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.858446 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.858446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.858454 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.860311 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.860801 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.860802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.860803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.860880 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.861156 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.861157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.861157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.861302 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.861346 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.861704 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.861786 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:08:16.861909 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.851037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:32823 (local address 127.0.203.60:42511)
0620 19:08:16.851075 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.851077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.851156 (+ 79us) server_negotiation.cc:407] Connection header received
0620 19:08:16.851585 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.851586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.851601 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.851631 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.852523 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.853320 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.854334 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.854625 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.854690 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.859029 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.859031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.859031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.859039 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.860763 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.861116 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.861117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.861117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.861169 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.861382 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.861382 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.861383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.861500 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.861539 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.861746 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.861826 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:08:16.862236 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.850713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:60403 (local address 127.0.203.61:41813)
0620 19:08:16.850760 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.850762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.850926 (+ 164us) server_negotiation.cc:407] Connection header received
0620 19:08:16.851584 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.851586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.851598 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.851631 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.852564 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.853109 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.854514 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.854747 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.854813 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.859071 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.859073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.859073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.859081 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:16.860976 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.861388 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.861389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.861389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.861442 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:16.861625 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:16.861625 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:16.861626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:16.861755 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:16.861796 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.862010 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.862092 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:08:16.862414 8016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.850637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.2:44997)
0620 19:08:16.850877 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.850885 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.851465 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.851744 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.851745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.852154 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.852456 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.852458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.853570 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.853570 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.854397 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.854398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.854890 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.854890 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.854947 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.855397 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.855425 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.858325 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.860405 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.860411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.860413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.860708 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.860955 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.860956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.860956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.861082 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.861392 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.861398 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.861549 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.861994 (+ 445us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.862195 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":6,"thread_start_us":85,"threads_started":1}
I20260620 19:08:16.862607 8015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.850637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:41813 (local address 127.0.203.2:60403)
0620 19:08:16.850791 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.850833 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.851464 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.851724 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.851728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.852172 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.852437 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.852443 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.853209 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.853210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.854396 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.854398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.854812 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.854813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.854877 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.855633 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.855640 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.858974 (+ 3334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.861066 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.861067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.861068 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.861308 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.861508 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.861508 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.861509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.861564 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.861848 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.861849 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.861902 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.862327 (+ 425us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.862443 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":21,"thread_start_us":85,"threads_started":1}
I20260620 19:08:16.863051 8013 heartbeater.cc:347] Connected to a master server at 127.0.203.61:41813
I20260620 19:08:16.863169 8013 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:16.863250 8017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.850940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:42511 (local address 127.0.203.2:32823)
0620 19:08:16.851073 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.851081 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.851473 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.852121 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.852122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.852262 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.852437 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.852443 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.853391 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.853392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.854224 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.854227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.854705 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.854706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.854839 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.855457 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.855461 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.858923 (+ 3462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:16.860839 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.860840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.860840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.861034 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.861241 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:16.861242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:16.861242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:16.861289 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:16.861584 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:16.861585 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:16.861635 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.862744 (+ 1109us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.863092 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":36,"thread_start_us":61,"threads_started":1}
I20260620 19:08:16.863379 8013 heartbeater.cc:511] Master 127.0.203.61:41813 requested a full tablet report, sending...
I20260620 19:08:16.863672 8012 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36095
I20260620 19:08:16.863765 8012 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:16.863838 8012 heartbeater.cc:511] Master 127.0.203.62:36095 requested a full tablet report, sending...
I20260620 19:08:16.863838 7509 ts_manager.cc:194] Registered new tserver with Master: 73d53fd8b46644c4a48293779879f814 (127.0.203.2:42711)
I20260620 19:08:16.863971 8011 heartbeater.cc:347] Connected to a master server at 127.0.203.60:42511
I20260620 19:08:16.864013 8011 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:16.864070 8011 heartbeater.cc:511] Master 127.0.203.60:42511 requested a full tablet report, sending...
I20260620 19:08:16.864343 7695 ts_manager.cc:194] Registered new tserver with Master: 73d53fd8b46644c4a48293779879f814 (127.0.203.2:42711)
I20260620 19:08:16.865118 6911 ts_manager.cc:194] Re-registered known tserver with Master: 73d53fd8b46644c4a48293779879f814 (127.0.203.2:42711)
I20260620 19:08:16.865489 6911 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:44997
I20260620 19:08:16.865871 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:41697
--local_ip_for_outbound_sockets=127.0.203.3
--tserver_master_addrs=127.0.203.62:36095
--webserver_port=37665
--webserver_interface=127.0.203.3
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.3.keytab
--principal=oryx/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.203.62:36095,127.0.203.61:41813
--tserver_master_addrs=127.0.203.62:36095,127.0.203.61:41813,127.0.203.60:42511 with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:16.946883 8018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:16.947083 8018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:16.947108 8018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:16.948283 8018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:16.948325 8018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:16.948352 8018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:08:16.949767 8018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/krb5kdc/oryx_127.0.203.3.keytab
--principal=oryx/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:41697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=37665
--tserver_master_addrs=127.0.203.62:36095,127.0.203.61:41813,127.0.203.60:42511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:16.949941 8018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:16.950114 8018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:16.955206 8023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:16.955235 8018 server_base.cc:1034] running on GCE node
W20260620 19:08:16.955188 8024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:16.955188 8026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:16.955576 8018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:16.955842 8018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:16.957013 8018 hybrid_clock.cc:648] HybridClock initialized: now 1781982496956999 us; error 28 us; skew 500 ppm
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:16.959524 8018 init.cc:377] Logged in from keytab as oryx/127.0.203.3@KRBTEST.COM (short username oryx)
I20260620 19:08:16.959964 8018 webserver.cc:460] Webserver started at http://127.0.203.3:37665/ using document root <none> and password file <none>
I20260620 19:08:16.960151 8018 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:16.960201 8018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:16.961494 8018 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:16.962018 8033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.962189 8018 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.962273 8018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "ec2d760a63644e13bde30b79f7913d16"
format_stamp: "Formatted at 2026-06-20 19:08:15 on dist-test-slave-wd30"
server_key: "20f3b0908a543f7a9bb49b72c399ac2d"
server_key_iv: "dce982afe249801e53e634b11c0f9910"
server_key_version: "encryptionkey@0"
I20260620 19:08:16.962582 8018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:16.972031 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.969410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37668 (local address 127.0.203.60:42511)
0620 19:08:16.969473 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.969475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.969486 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:08:16.969542 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.969543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.969564 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.969643 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:16.969781 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.970288 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.971305 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.971697 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.971785 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.971860 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:16.971922 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 19:08:16.971959 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:08:16.972031 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.969353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:42511 (local address 127.0.0.1:37668)
0620 19:08:16.969404 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.969418 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.969485 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.969647 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.969648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.969650 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:16.969706 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.969708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.970383 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.970384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.971214 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.971216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.971788 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.971789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.971852 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.971856 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:08:16.971916 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 19:08:16.971961 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 19:08:16.972388 7705 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 19:08:16.972540 7734 sys_catalog.cc:455] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b58adcd02d204af8810c43fa357d0ffa. Latest consensus state: current_term: 1 leader_uuid: "b58adcd02d204af8810c43fa357d0ffa" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: NON_VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: true } } }
I20260620 19:08:16.972668 7734 sys_catalog.cc:458] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [sys.catalog]: This master's current role is: LEARNER
I20260620 19:08:16.983387 8018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:16.985378 8018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:16.985539 8018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:16.985781 8018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:16.986091 8018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:16.986145 8018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.986184 8018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:16.986212 8018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:16.992100 8018 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:41697
I20260620 19:08:16.992201 8146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:41697 every 8 connection(s)
I20260620 19:08:16.992537 8018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.60@KRBTEST.COM
Jun 20 19:08:16 dist-test-slave-wd30 krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.61@KRBTEST.COM
I20260620 19:08:17.001255 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 8018
I20260620 19:08:17.003772 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.993762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:36957 (local address 127.0.203.60:42511)
0620 19:08:16.993808 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.993810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.994369 (+ 559us) server_negotiation.cc:407] Connection header received
0620 19:08:16.994394 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.994395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.994409 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.994460 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.995276 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.995759 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.997126 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.997422 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.997463 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:17.000315 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:17.000317 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:17.000317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:17.000325 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:17.002103 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:17.002655 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:17.002656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:17.002656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:17.002713 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:17.002993 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:17.002994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:17.002995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:17.003150 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:17.003195 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:17.003587 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 19:08:17.003673 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:08:17.003767 6966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.993695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:51085 (local address 127.0.203.62:36095)
0620 19:08:16.993756 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.993757 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.993768 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:08:16.994321 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.994322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.994352 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.994403 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.995298 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.995783 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.997115 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.997319 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.997395 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.999841 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:16.999842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:16.999843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:16.999850 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:17.002383 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:17.002843 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:17.002844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:17.002844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:17.002906 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:17.003109 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:17.003111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:17.003111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:17.003288 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:17.003351 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:17.003590 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 19:08:17.003679 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:08:17.004051 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.993291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:41813 (local address 127.0.203.3:34929)
0620 19:08:16.993470 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.993511 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.994108 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.994359 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.994364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.994939 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.995180 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.995186 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.996680 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.996681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.997462 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.997464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.997902 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.997903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.997984 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.998740 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.998746 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:17.000717 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:17.002427 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:17.002428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:17.002428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:17.002878 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:17.003214 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:17.003214 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:17.003215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:17.003271 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:17.003549 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:17.003550 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:17.003585 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 19:08:17.003656 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:08:17.003824 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260620 19:08:17.004189 8151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.993575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:42511 (local address 127.0.203.3:36957)
0620 19:08:16.994287 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.994296 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.994345 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.994500 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.994500 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.994946 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.995179 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.995185 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.995884 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.995885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.997030 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.997032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.997539 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.997539 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.997605 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.998269 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.998274 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:17.000216 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:17.002198 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:17.002204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:17.002206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:17.002567 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:17.002787 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:17.002787 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:17.002788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:17.002913 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:17.003242 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:17.003248 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:17.003431 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 19:08:17.003931 (+ 500us) client_negotiation.cc:240] Negotiation successful
0620 19:08:17.004082 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":81,"thread_start_us":57,"threads_started":1}
I20260620 19:08:17.004397 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.993292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.3:51085)
0620 19:08:16.993678 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:16.993688 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:16.994116 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:16.994469 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:16.994470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:16.994946 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:16.995179 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.995185 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.995888 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.995889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.997029 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:16.997031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.997398 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.997399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:16.997560 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:16.998192 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:16.998200 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:16.999749 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:17.002467 (+ 2718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:17.002468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:17.002468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:17.002761 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:17.002982 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:17.002982 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:17.002983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:17.003035 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:17.003377 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:17.003378 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:17.003437 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 19:08:17.003916 (+ 479us) client_negotiation.cc:240] Negotiation successful
0620 19:08:17.004019 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":91,"thread_start_us":107,"threads_started":1}
I20260620 19:08:17.004555 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:16.993252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:34929 (local address 127.0.203.61:41813)
0620 19:08:16.993305 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:16.993308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:16.993606 (+ 298us) server_negotiation.cc:407] Connection header received
0620 19:08:16.994213 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:16.994214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:16.994224 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:16.994277 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:16.995873 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.996596 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.997538 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:16.997830 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:16.997884 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:17.000819 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:17.000820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:17.000821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:17.000828 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:17.002330 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:17.003021 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:17.003022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:17.003022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:17.003108 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:17.003342 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:17.003343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:17.003343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:17.003463 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:17.003525 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:17.004368 (+ 843us) server_negotiation.cc:299] Negotiation successful
0620 19:08:17.004444 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:08:17.004554 8147 heartbeater.cc:347] Connected to a master server at 127.0.203.60:42511
I20260620 19:08:17.004639 8147 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:17.004824 8147 heartbeater.cc:511] Master 127.0.203.60:42511 requested a full tablet report, sending...
I20260620 19:08:17.004897 8148 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36095
I20260620 19:08:17.004956 8148 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:17.005036 8148 heartbeater.cc:511] Master 127.0.203.62:36095 requested a full tablet report, sending...
I20260620 19:08:17.005221 8149 heartbeater.cc:347] Connected to a master server at 127.0.203.61:41813
I20260620 19:08:17.005264 8149 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:17.005293 6911 ts_manager.cc:194] Re-registered known tserver with Master: ec2d760a63644e13bde30b79f7913d16 (127.0.203.3:41697)
I20260620 19:08:17.005317 8149 heartbeater.cc:511] Master 127.0.203.61:41813 requested a full tablet report, sending...
I20260620 19:08:17.005333 7695 ts_manager.cc:194] Registered new tserver with Master: ec2d760a63644e13bde30b79f7913d16 (127.0.203.3:41697)
I20260620 19:08:17.005575 7509 ts_manager.cc:194] Registered new tserver with Master: ec2d760a63644e13bde30b79f7913d16 (127.0.203.3:41697)
I20260620 19:08:17.005810 6911 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:51085
I20260620 19:08:17.012687 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:17.003489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37674 (local address 127.0.203.60:42511)
0620 19:08:17.003875 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:17.003877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:17.005154 (+ 1277us) server_negotiation.cc:407] Connection header received
0620 19:08:17.005249 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:17.005250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:17.005260 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:17.005288 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:17.006034 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.006700 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.007798 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.008066 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.008124 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:17.009599 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:17.009600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:17.009600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:17.009608 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:17.011355 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:17.011758 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:17.011759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:17.011760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:17.011810 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:17.012016 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:17.012018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:17.012018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:17.012154 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:17.012261 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:17.012467 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 19:08:17.012539 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":56,"threads_started":1}
I20260620 19:08:17.013398 8159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:17.002906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.0.203.62:36095)
0620 19:08:17.003722 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:17.003724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:17.006517 (+ 2793us) server_negotiation.cc:407] Connection header received
0620 19:08:17.006542 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:17.006542 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:17.006553 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:17.006630 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:17.007227 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.007775 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.008709 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.008987 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.009064 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:17.010333 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:17.010335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:17.010335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:17.010344 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:17.012160 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:17.012520 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:17.012521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:17.012521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:17.012577 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:17.012793 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:17.012794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:17.012794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:17.012933 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:17.013026 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:17.013197 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 19:08:17.013279 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":82,"threads_started":1}
I20260620 19:08:17.013995 8158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:17.002752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52084 (local address 127.0.203.61:41813)
0620 19:08:17.002982 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:17.002984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:17.005927 (+ 2943us) server_negotiation.cc:407] Connection header received
0620 19:08:17.005954 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:17.005954 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:17.005966 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:17.006030 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:17.007365 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.007991 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.008882 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.009258 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.009341 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:17.010812 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:17.010813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:17.010814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:17.010820 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:17.012492 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:17.012877 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:17.012878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:17.012879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:17.012928 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:17.013276 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:17.013277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:17.013277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:17.013411 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:17.013515 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:17.013803 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 19:08:17.013894 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":52,"threads_started":1}
I20260620 19:08:17.014981 6911 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:08:17.016557 6911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:08:17.017295 7470 sys_catalog.cc:455] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b58adcd02d204af8810c43fa357d0ffa" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: NON_VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: true } } }
I20260620 19:08:17.017431 7470 sys_catalog.cc:458] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:17.018563 7661 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa: attempting to promote NON_VOTER d780f2401c23420e90ff49507df4851a to VOTER
I20260620 19:08:17.018688 7661 consensus_queue.cc:237] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: false } }
I20260620 19:08:17.019234 7519 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 1 FOLLOWER]: Refusing update from remote peer b58adcd02d204af8810c43fa357d0ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 19:08:17.019479 7471 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 19:08:17.019713 7704 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 1 LEARNER]: Refusing update from remote peer b58adcd02d204af8810c43fa357d0ffa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 19:08:17.019932 7470 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 19:08:17.020383 7471 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d780f2401c23420e90ff49507df4851a (127.0.203.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: false } } }
I20260620 19:08:17.021113 7471 sys_catalog.cc:455] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b58adcd02d204af8810c43fa357d0ffa" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: false } } }
I20260620 19:08:17.021173 8170 sys_catalog.cc:455] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b58adcd02d204af8810c43fa357d0ffa" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: false } } }
I20260620 19:08:17.021276 8170 sys_catalog.cc:458] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:17.021360 7519 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d780f2401c23420e90ff49507df4851a (127.0.203.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: false } } }
I20260620 19:08:17.021456 8170 sys_catalog.cc:455] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b58adcd02d204af8810c43fa357d0ffa" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: false } } }
I20260620 19:08:17.021525 8170 sys_catalog.cc:458] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:17.021186 7471 sys_catalog.cc:458] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:17.021693 7704 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d780f2401c23420e90ff49507df4851a (127.0.203.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: false } } }
I20260620 19:08:17.022337 7658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b58adcd02d204af8810c43fa357d0ffa" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: false } } }
I20260620 19:08:17.022436 7658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:08:17.022747 7734 sys_catalog.cc:455] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b58adcd02d204af8810c43fa357d0ffa" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: false } } }
I20260620 19:08:17.022830 7734 sys_catalog.cc:458] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:08:17.025939 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:17.022332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:42711 (local address 127.0.0.1:50030)
0620 19:08:17.022503 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:17.022515 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:17.022557 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:17.023390 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:17.023391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:17.023392 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:17.023436 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:17.023438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.024277 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.024279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:17.024969 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:17.024971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.025725 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.025726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:17.025794 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:17.025803 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 19:08:17.025828 (+ 25us) client_negotiation.cc:240] Negotiation successful
0620 19:08:17.025858 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260620 19:08:17.026330 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:17.022459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50030 (local address 127.0.203.2:42711)
0620 19:08:17.023214 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:17.023219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:17.023230 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:08:17.023270 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:17.023272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:17.023290 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:17.023393 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:17.023498 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.024180 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.025057 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.025623 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.026104 (+ 481us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:17.026194 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:17.026219 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 19:08:17.026255 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":88,"threads_started":1}
I20260620 19:08:17.026943 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:17.022347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:40925 (local address 127.0.0.1:33162)
0620 19:08:17.023114 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:17.023143 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:17.023191 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:17.023459 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:17.023460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:17.023462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:17.023498 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:17.023499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.024704 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.024705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:17.025827 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:17.025829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.026706 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.026707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:17.026770 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:17.026773 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:08:17.026837 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 19:08:17.026876 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":71,"threads_started":1}
I20260620 19:08:17.026973 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:17.022783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.0.203.1:40925)
0620 19:08:17.023111 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:17.023135 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:17.023250 (+ 115us) server_negotiation.cc:407] Connection header received
0620 19:08:17.023307 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:17.023309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:17.023324 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:17.023395 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:17.023551 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.024614 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.025910 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.026634 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.026704 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:17.026768 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:17.026856 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 19:08:17.026893 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":96,"threads_started":1}
I20260620 19:08:17.026965 7945 tablet_service.cc:1467] Processing CreateTablet for tablet ade2667c7b5d41a2a56bfd005d1bff66 (DEFAULT_TABLE table=test-table [id=2c03ea9786b34a6799a168bd9618fdb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:17.027346 7945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade2667c7b5d41a2a56bfd005d1bff66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:17.027436 8172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:17.022505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:41697 (local address 127.0.0.1:36014)
0620 19:08:17.023675 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:17.023684 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:17.023729 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:17.023951 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:17.023952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:17.023954 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:17.023991 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:17.023993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.025142 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.025143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:17.026231 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:17.026234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.027194 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.027195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:17.027258 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:17.027262 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:08:17.027318 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 19:08:17.027359 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1154,"thread_start_us":74,"threads_started":1}
I20260620 19:08:17.027635 8176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:17.022851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36014 (local address 127.0.203.3:41697)
0620 19:08:17.023624 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:17.023629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:17.023779 (+ 150us) server_negotiation.cc:407] Connection header received
0620 19:08:17.023820 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:17.023822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:17.023837 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:17.023899 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:17.024053 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.025057 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.026317 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.027074 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.027189 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:17.027256 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:17.027528 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 19:08:17.027567 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":76,"threads_started":1}
I20260620 19:08:17.027877 7808 tablet_service.cc:1467] Processing CreateTablet for tablet ade2667c7b5d41a2a56bfd005d1bff66 (DEFAULT_TABLE table=test-table [id=2c03ea9786b34a6799a168bd9618fdb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:17.028172 7808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade2667c7b5d41a2a56bfd005d1bff66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:17.030230 8177 tablet_bootstrap.cc:492] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814: Bootstrap starting.
I20260620 19:08:17.030771 8178 tablet_bootstrap.cc:492] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa: Bootstrap starting.
I20260620 19:08:17.030905 8177 tablet_bootstrap.cc:654] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:17.031256 8177 log.cc:824] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:17.031445 8178 tablet_bootstrap.cc:654] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:17.031473 8081 tablet_service.cc:1467] Processing CreateTablet for tablet ade2667c7b5d41a2a56bfd005d1bff66 (DEFAULT_TABLE table=test-table [id=2c03ea9786b34a6799a168bd9618fdb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:17.031785 8178 log.cc:824] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:17.031809 8081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade2667c7b5d41a2a56bfd005d1bff66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:17.032299 8177 tablet_bootstrap.cc:492] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814: No bootstrap required, opened a new log
I20260620 19:08:17.032373 8177 ts_tablet_manager.cc:1397] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:08:17.032487 8178 tablet_bootstrap.cc:492] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa: No bootstrap required, opened a new log
I20260620 19:08:17.033831 8178 ts_tablet_manager.cc:1397] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 19:08:17.035365 8177 raft_consensus.cc:348] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 } } peers { permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 } }
I20260620 19:08:17.035533 8177 raft_consensus.cc:374] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:17.035573 8177 raft_consensus.cc:729] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d53fd8b46644c4a48293779879f814, State: Initialized, Role: FOLLOWER
I20260620 19:08:17.035692 8177 consensus_queue.cc:260] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 } } peers { permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 } }
I20260620 19:08:17.036031 8012 heartbeater.cc:503] Master 127.0.203.62:36095 was elected leader, sending a full tablet report...
I20260620 19:08:17.035946 8178 raft_consensus.cc:348] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 } } peers { permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 } }
I20260620 19:08:17.036303 8178 raft_consensus.cc:374] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:17.036343 8178 raft_consensus.cc:729] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a241202a70f420ca87abfce500521fa, State: Initialized, Role: FOLLOWER
I20260620 19:08:17.036381 8177 ts_tablet_manager.cc:1428] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 19:08:17.036477 8178 consensus_queue.cc:260] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 } } peers { permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 } }
I20260620 19:08:17.036725 8178 ts_tablet_manager.cc:1428] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:08:17.037189 7875 heartbeater.cc:503] Master 127.0.203.62:36095 was elected leader, sending a full tablet report...
I20260620 19:08:17.037503 8181 tablet_bootstrap.cc:492] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16: Bootstrap starting.
I20260620 19:08:17.038041 8181 tablet_bootstrap.cc:654] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:17.038259 8181 log.cc:824] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:17.039109 8181 tablet_bootstrap.cc:492] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16: No bootstrap required, opened a new log
I20260620 19:08:17.039216 8181 ts_tablet_manager.cc:1397] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:08:17.040593 8181 raft_consensus.cc:348] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 } } peers { permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 } }
I20260620 19:08:17.040728 8181 raft_consensus.cc:374] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:17.040771 8181 raft_consensus.cc:729] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2d760a63644e13bde30b79f7913d16, State: Initialized, Role: FOLLOWER
I20260620 19:08:17.040881 8181 consensus_queue.cc:260] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 } } peers { permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 } }
I20260620 19:08:17.041121 8181 ts_tablet_manager.cc:1428] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:08:17.041189 8148 heartbeater.cc:503] Master 127.0.203.62:36095 was elected leader, sending a full tablet report...
W20260620 19:08:17.100759 8014 tablet.cc:2369] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:08:17.154975 8183 raft_consensus.cc:482] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:08:17.155097 8183 raft_consensus.cc:504] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 } } peers { permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 } }
I20260620 19:08:17.155521 8183 leader_election.cc:288] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec2d760a63644e13bde30b79f7913d16 (127.0.203.3:41697), 73d53fd8b46644c4a48293779879f814 (127.0.203.2:42711)
I20260620 19:08:17.159307 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:17.155833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:37737 (local address 127.0.203.2:42711)
0620 19:08:17.155882 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:17.155884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:17.155903 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:08:17.155929 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:17.155930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:17.155943 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:17.155995 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:17.156144 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.156890 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.158375 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.158935 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.159002 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:17.159012 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:17.159199 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 19:08:17.159237 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 19:08:17.159307 8176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:17.155766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:42119 (local address 127.0.203.3:41697)
0620 19:08:17.155817 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:17.155820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:17.155843 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:08:17.155870 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:17.155871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:17.155884 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:17.155935 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:17.156148 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.156889 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.158374 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.158926 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.159002 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:17.159012 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:17.159195 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 19:08:17.159236 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 19:08:17.159325 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:17.155721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:42711 (local address 127.0.203.1:37737)
0620 19:08:17.155786 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:17.155796 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:17.155837 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:17.156011 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:17.156011 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:17.156013 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:17.156064 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:17.156065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.156975 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.156976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:17.158280 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:17.158283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.159013 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.159013 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:17.159098 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:17.159102 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:08:17.159177 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 19:08:17.159229 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":90}
I20260620 19:08:17.159343 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:17.155683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:41697 (local address 127.0.203.1:42119)
0620 19:08:17.155738 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:17.155750 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:17.155794 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:17.155954 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:17.155957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:17.155959 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:17.156064 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:17.156065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.156977 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.156977 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:17.158280 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:17.158283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.159013 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.159013 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:17.159104 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:17.159106 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:08:17.159177 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:08:17.159229 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":68}
I20260620 19:08:17.159708 7965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade2667c7b5d41a2a56bfd005d1bff66" candidate_uuid: "6a241202a70f420ca87abfce500521fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d53fd8b46644c4a48293779879f814" is_pre_election: true
I20260620 19:08:17.159861 7965 raft_consensus.cc:2454] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a241202a70f420ca87abfce500521fa in term 0.
I20260620 19:08:17.160102 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade2667c7b5d41a2a56bfd005d1bff66" candidate_uuid: "6a241202a70f420ca87abfce500521fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2d760a63644e13bde30b79f7913d16" is_pre_election: true
I20260620 19:08:17.160126 7761 leader_election.cc:302] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a241202a70f420ca87abfce500521fa, 73d53fd8b46644c4a48293779879f814; no voters:
I20260620 19:08:17.160254 8101 raft_consensus.cc:2454] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a241202a70f420ca87abfce500521fa in term 0.
I20260620 19:08:17.160276 8183 raft_consensus.cc:2781] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:08:17.160322 8183 raft_consensus.cc:482] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:08:17.160364 8183 raft_consensus.cc:3037] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:17.161062 8183 raft_consensus.cc:504] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 } } peers { permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 } }
I20260620 19:08:17.161209 8183 leader_election.cc:288] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [CANDIDATE]: Term 1 election: Requested vote from peers ec2d760a63644e13bde30b79f7913d16 (127.0.203.3:41697), 73d53fd8b46644c4a48293779879f814 (127.0.203.2:42711)
I20260620 19:08:17.161345 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade2667c7b5d41a2a56bfd005d1bff66" candidate_uuid: "6a241202a70f420ca87abfce500521fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2d760a63644e13bde30b79f7913d16"
I20260620 19:08:17.161341 7965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade2667c7b5d41a2a56bfd005d1bff66" candidate_uuid: "6a241202a70f420ca87abfce500521fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d53fd8b46644c4a48293779879f814"
I20260620 19:08:17.161442 8101 raft_consensus.cc:3037] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:17.161437 7965 raft_consensus.cc:3037] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:17.162134 8101 raft_consensus.cc:2454] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a241202a70f420ca87abfce500521fa in term 1.
I20260620 19:08:17.162137 7965 raft_consensus.cc:2454] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a241202a70f420ca87abfce500521fa in term 1.
I20260620 19:08:17.162370 7762 leader_election.cc:302] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a241202a70f420ca87abfce500521fa, ec2d760a63644e13bde30b79f7913d16; no voters:
I20260620 19:08:17.162492 8183 raft_consensus.cc:2781] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:17.162672 8183 raft_consensus.cc:686] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [term 1 LEADER]: Becoming Leader. State: Replica: 6a241202a70f420ca87abfce500521fa, State: Running, Role: LEADER
I20260620 19:08:17.162770 8183 consensus_queue.cc:237] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 } } peers { permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 } }
I20260620 19:08:17.163842 6911 catalog_manager.cc:5495] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a241202a70f420ca87abfce500521fa (127.0.203.1). New cstate: current_term: 1 leader_uuid: "6a241202a70f420ca87abfce500521fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 } health_report { overall_health: UNKNOWN } } }
I20260620 19:08:17.204432 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:17.202218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.0.203.1:40925)
0620 19:08:17.202291 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:17.202292 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:17.202303 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:08:17.202338 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:17.202338 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:17.202359 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:17.202420 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:08:17.202519 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.202972 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.203692 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.203881 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.203935 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:17.204095 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:08:17.204190 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:08:17.204264 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:17.204283 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 19:08:17.204309 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:08:17.206699 8101 raft_consensus.cc:1261] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16 [term 1 FOLLOWER]: Refusing update from remote peer 6a241202a70f420ca87abfce500521fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:17.206776 7965 raft_consensus.cc:1261] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814 [term 1 FOLLOWER]: Refusing update from remote peer 6a241202a70f420ca87abfce500521fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:17.207103 8186 consensus_queue.cc:1035] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:17.207247 8183 consensus_queue.cc:1035] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:08:17.207926 7877 tablet.cc:2369] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:08:17.210273 8189 mvcc.cc:205] Tried to move back new op lower bound from 7299000308555857920 to 7299000308379013120. Current Snapshot: MvccSnapshot[applied={T|T < 7299000308555857920}]
I20260620 19:08:17.210436 8191 mvcc.cc:205] Tried to move back new op lower bound from 7299000308555857920 to 7299000308379013120. Current Snapshot: MvccSnapshot[applied={T|T < 7299000308555857920}]
I20260620 19:08:17.213743 6911 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:17.213840 6911 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:17.215049 6911 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b58adcd02d204af8810c43fa357d0ffa: Sending DeleteTablet for 3 replicas of tablet ade2667c7b5d41a2a56bfd005d1bff66
I20260620 19:08:17.215422 7945 tablet_service.cc:1514] Processing DeleteTablet for tablet ade2667c7b5d41a2a56bfd005d1bff66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:17 UTC) from {username='oryx', principal='oryx/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:50030
I20260620 19:08:17.215888 8081 tablet_service.cc:1514] Processing DeleteTablet for tablet ade2667c7b5d41a2a56bfd005d1bff66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:17 UTC) from {username='oryx', principal='oryx/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:36014
I20260620 19:08:17.215989 8199 tablet_replica.cc:330] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814: stopping tablet replica
I20260620 19:08:17.216005 7808 tablet_service.cc:1514] Processing DeleteTablet for tablet ade2667c7b5d41a2a56bfd005d1bff66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:17 UTC) from {username='oryx', principal='oryx/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:33162
I20260620 19:08:17.216081 8199 raft_consensus.cc:2229] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:17.216172 8199 raft_consensus.cc:2258] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:17.216208 8202 tablet_replica.cc:330] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa: stopping tablet replica
I20260620 19:08:17.216287 8202 raft_consensus.cc:2229] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:08:17.216409 8202 raft_consensus.cc:2258] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:17.216626 8199 ts_tablet_manager.cc:1905] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:17.216835 8202 ts_tablet_manager.cc:1905] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:17.217329 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 6879
I20260620 19:08:17.218286 8199 ts_tablet_manager.cc:1918] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:17.218348 8199 log.cc:1196] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/ade2667c7b5d41a2a56bfd005d1bff66
I20260620 19:08:17.218612 8199 ts_tablet_manager.cc:1939] T ade2667c7b5d41a2a56bfd005d1bff66 P 73d53fd8b46644c4a48293779879f814: Deleting consensus metadata
I20260620 19:08:17.219514 8202 ts_tablet_manager.cc:1918] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:17.219620 8202 log.cc:1196] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-0/wal/wals/ade2667c7b5d41a2a56bfd005d1bff66
I20260620 19:08:17.219871 8202 ts_tablet_manager.cc:1939] T ade2667c7b5d41a2a56bfd005d1bff66 P 6a241202a70f420ca87abfce500521fa: Deleting consensus metadata
I20260620 19:08:17.223465 8201 tablet_replica.cc:330] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16: stopping tablet replica
I20260620 19:08:17.223639 8201 raft_consensus.cc:2229] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:17.223788 8201 raft_consensus.cc:2258] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 19:08:17.224030 7763 connection.cc:535] server connection from 127.0.0.1:33162 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 19:08:17.224231 7901 connection.cc:535] server connection from 127.0.0.1:50030 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:08:17.224423 8201 ts_tablet_manager.cc:1905] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:17.226922 8201 ts_tablet_manager.cc:1918] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:17.227061 8201 log.cc:1196] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/ade2667c7b5d41a2a56bfd005d1bff66
I20260620 19:08:17.227430 8201 ts_tablet_manager.cc:1939] T ade2667c7b5d41a2a56bfd005d1bff66 P ec2d760a63644e13bde30b79f7913d16: Deleting consensus metadata
W20260620 19:08:17.227813 8035 connection.cc:441] server connection from 127.0.0.1:36014 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36014 (request call id 1) could send its response
I20260620 19:08:17.231854 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:17.215879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37678 (local address 127.0.203.60:42511)
0620 19:08:17.215914 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:17.215917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:17.223333 (+ 7416us) server_negotiation.cc:407] Connection header received
0620 19:08:17.223389 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:17.223391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:17.223403 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:17.223462 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:17.224331 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.225084 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.226197 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.226556 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.226628 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:17.228218 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:17.228220 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:17.228220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:17.228227 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:17.229988 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:17.230365 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:17.230366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:17.230367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:17.230425 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:17.230625 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:17.230626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:17.230626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:17.231041 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:17.231163 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:17.231642 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 19:08:17.231735 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:08:17.235733 8161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:17.226524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37684 (local address 127.0.203.60:42511)
0620 19:08:17.226565 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:17.226567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:17.228059 (+ 1492us) server_negotiation.cc:407] Connection header received
0620 19:08:17.228159 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:17.228160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:17.228172 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:17.228209 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:17.228849 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.229624 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.230676 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.230994 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.231032 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:17.232775 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:17.232776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:17.232777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:17.232783 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:17.234449 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:17.234859 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:17.234860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:17.234860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:17.234921 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:17.235152 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:17.235152 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:17.235153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:17.235293 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:17.235390 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:17.235525 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 19:08:17.235613 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:08:17.235921 8158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:17.226425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52086 (local address 127.0.203.61:41813)
0620 19:08:17.226719 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:17.226721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:17.226729 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:08:17.226751 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:17.226752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:17.226763 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:17.226819 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:17.227615 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.228403 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.230013 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:17.230332 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:17.230375 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:17.231993 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:17.231994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:17.231994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:17.232001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:17.233673 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:17.234893 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:17.234895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:17.234895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:17.234946 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:17.235331 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:17.235332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:17.235333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:17.235461 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:17.235534 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:17.235736 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 19:08:17.235828 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265}
I20260620 19:08:18.211772 8209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.211356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.2:60747)
0620 19:08:18.211627 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.211711 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":63,"threads_started":1}
W20260620 19:08:18.212038 8012 heartbeater.cc:644] Failed to heartbeat to 127.0.203.62:36095 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
I20260620 19:08:18.212237 8210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.212024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.1:55875)
0620 19:08:18.212136 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.212195 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":49,"threads_started":1}
I20260620 19:08:18.212289 8211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.212015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.3:40353)
0620 19:08:18.212156 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.212243 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
W20260620 19:08:18.212405 7875 heartbeater.cc:644] Failed to heartbeat to 127.0.203.62:36095 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
W20260620 19:08:18.212459 8148 heartbeater.cc:644] Failed to heartbeat to 127.0.203.62:36095 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
I20260620 19:08:18.715814 8212 raft_consensus.cc:482] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b58adcd02d204af8810c43fa357d0ffa)
I20260620 19:08:18.715999 8212 raft_consensus.cc:504] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: false } }
I20260620 19:08:18.716281 8212 leader_election.cc:288] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b58adcd02d204af8810c43fa357d0ffa (127.0.203.62:36095), 324f3d9c6e264f09bd8ad5313baee734 (127.0.203.61:41813)
I20260620 19:08:18.716827 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.716455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.0.1:42926)
0620 19:08:18.716735 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.716783 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":3,"thread_start_us":45,"threads_started":1}
W20260620 19:08:18.716971 7681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
I20260620 19:08:18.717094 8216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b58adcd02d204af8810c43fa357d0ffa)
I20260620 19:08:18.717204 8216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: false } }
I20260620 19:08:18.717468 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.717357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.0.1:42946)
0620 19:08:18.717412 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.717442 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 19:08:18.717623 7681 leader_election.cc:334] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b58adcd02d204af8810c43fa357d0ffa (127.0.203.62:36095): Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
I20260620 19:08:18.717707 8216 leader_election.cc:288] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b58adcd02d204af8810c43fa357d0ffa (127.0.203.62:36095), d780f2401c23420e90ff49507df4851a (127.0.203.60:42511)
I20260620 19:08:18.717831 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.717519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.0.1:42948)
0620 19:08:18.717739 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.717790 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":52,"threads_started":1}
W20260620 19:08:18.717924 7495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
I20260620 19:08:18.718361 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.718270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.0.1:42958)
0620 19:08:18.718306 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.718336 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 19:08:18.718513 7495 leader_election.cc:334] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b58adcd02d204af8810c43fa357d0ffa (127.0.203.62:36095): Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
I20260620 19:08:18.720206 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.716578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52096 (local address 127.0.203.61:41813)
0620 19:08:18.716811 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:18.716814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:18.717090 (+ 276us) server_negotiation.cc:407] Connection header received
0620 19:08:18.717117 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:18.717118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:18.717129 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:18.717348 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:18.717389 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:18.718208 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:18.719443 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:18.719895 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:18.719988 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:18.720000 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:18.720108 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 19:08:18.720142 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":72,"threads_started":1}
I20260620 19:08:18.720209 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.716455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:41813 (local address 127.0.0.1:52096)
0620 19:08:18.716861 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.716924 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:18.716994 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:18.717216 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:18.717217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:18.717219 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:18.717284 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:18.717286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:18.718280 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:18.718281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:18.719365 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:18.719367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:18.719979 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:18.719980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:18.720043 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:18.720047 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:08:18.720098 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:08:18.720144 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":51,"threads_started":1}
I20260620 19:08:18.720736 7519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d780f2401c23420e90ff49507df4851a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "324f3d9c6e264f09bd8ad5313baee734" is_pre_election: true
I20260620 19:08:18.720853 7519 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d780f2401c23420e90ff49507df4851a in term 1.
I20260620 19:08:18.720958 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.717730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37690 (local address 127.0.203.60:42511)
0620 19:08:18.717946 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:18.717948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:18.717954 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:08:18.717986 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:18.717987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:18.717998 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:18.718071 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:18.718246 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:18.718849 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:18.720072 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:18.720612 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:18.720690 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:18.720699 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:18.720854 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 19:08:18.720889 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":75,"threads_started":1}
I20260620 19:08:18.720954 8219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.717519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:42511 (local address 127.0.0.1:37690)
0620 19:08:18.717796 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.717805 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:18.717856 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:18.718093 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:18.718095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:18.718097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:18.718136 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:18.718137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:18.718948 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:18.718949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:18.719942 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:18.719944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:18.720692 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:18.720693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:18.720771 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:18.720774 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:08:18.720846 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 19:08:18.720888 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":2,"thread_start_us":53,"threads_started":1}
I20260620 19:08:18.721429 7680 leader_election.cc:302] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 324f3d9c6e264f09bd8ad5313baee734, d780f2401c23420e90ff49507df4851a; no voters: b58adcd02d204af8810c43fa357d0ffa
I20260620 19:08:18.721529 8212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:08:18.721589 8212 raft_consensus.cc:482] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 1 FOLLOWER]: Starting leader election (detected failure of leader b58adcd02d204af8810c43fa357d0ffa)
I20260620 19:08:18.721613 8212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:08:18.721607 7704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "324f3d9c6e264f09bd8ad5313baee734" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d780f2401c23420e90ff49507df4851a" is_pre_election: true
I20260620 19:08:18.722395 8212 raft_consensus.cc:504] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: false } }
I20260620 19:08:18.722527 7704 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 324f3d9c6e264f09bd8ad5313baee734 in current term 2: Already voted for candidate d780f2401c23420e90ff49507df4851a in this term.
I20260620 19:08:18.722543 8212 leader_election.cc:288] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [CANDIDATE]: Term 2 election: Requested vote from peers b58adcd02d204af8810c43fa357d0ffa (127.0.203.62:36095), 324f3d9c6e264f09bd8ad5313baee734 (127.0.203.61:41813)
I20260620 19:08:18.722694 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.722608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.0.1:42966)
0620 19:08:18.722642 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.722674 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 19:08:18.722697 7494 leader_election.cc:302] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 324f3d9c6e264f09bd8ad5313baee734; no voters: b58adcd02d204af8810c43fa357d0ffa, d780f2401c23420e90ff49507df4851a
I20260620 19:08:18.722846 8216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:08:18.722879 7519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d780f2401c23420e90ff49507df4851a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "324f3d9c6e264f09bd8ad5313baee734"
I20260620 19:08:18.723040 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.722873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.0.1:42976)
0620 19:08:18.722944 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.723016 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 19:08:18.723143 7681 leader_election.cc:334] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b58adcd02d204af8810c43fa357d0ffa (127.0.203.62:36095): Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
I20260620 19:08:18.723616 8216 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:08:18.724187 7519 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d780f2401c23420e90ff49507df4851a in term 2.
I20260620 19:08:18.724337 7680 leader_election.cc:302] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 324f3d9c6e264f09bd8ad5313baee734, d780f2401c23420e90ff49507df4851a; no voters: b58adcd02d204af8810c43fa357d0ffa
I20260620 19:08:18.724421 8212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:08:18.724568 8212 raft_consensus.cc:686] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [term 2 LEADER]: Becoming Leader. State: Replica: d780f2401c23420e90ff49507df4851a, State: Running, Role: LEADER
I20260620 19:08:18.724639 8212 consensus_queue.cc:237] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: false } }
I20260620 19:08:18.724983 8222 sys_catalog.cc:455] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d780f2401c23420e90ff49507df4851a. Latest consensus state: current_term: 2 leader_uuid: "d780f2401c23420e90ff49507df4851a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: false } } }
I20260620 19:08:18.725056 8222 sys_catalog.cc:458] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:18.725201 8224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:08:18.725617 8224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:08:18.725677 8224 catalog_manager.cc:1259] Loaded cluster ID: b794508261304e84b9d0314b8f040e46
I20260620 19:08:18.725694 8224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:08:18.725816 8224 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:18.725868 8224 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: Loaded TSK: 0
I20260620 19:08:18.725960 8224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:08:18.772132 7695 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:08:18.773754 7695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:08:18.774821 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.774706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.0.1:42990)
0620 19:08:18.774761 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.774795 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 19:08:18.774827 7519 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [term 2 FOLLOWER]: Refusing update from remote peer d780f2401c23420e90ff49507df4851a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 19:08:18.775163 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.775040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.0.1:43006)
0620 19:08:18.775087 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.775111 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 19:08:18.775084 8222 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [LEADER]: Connected to new peer: Peer: permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 19:08:18.775267 7681 consensus_peers.cc:487] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a -> Peer b58adcd02d204af8810c43fa357d0ffa (127.0.203.62:36095): Couldn't send request to peer b58adcd02d204af8810c43fa357d0ffa. Status: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 19:08:18.775803 8216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d780f2401c23420e90ff49507df4851a. Latest consensus state: current_term: 2 leader_uuid: "d780f2401c23420e90ff49507df4851a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: false } } }
I20260620 19:08:18.775911 8216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:08:18.776229 8225 mvcc.cc:205] Tried to move back new op lower bound from 7299000314979508224 to 7299000314776731648. Current Snapshot: MvccSnapshot[applied={T|T < 7299000314979508224}]
I20260620 19:08:18.776325 8212 sys_catalog.cc:455] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d780f2401c23420e90ff49507df4851a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: false } } }
I20260620 19:08:18.776419 8212 sys_catalog.cc:458] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:18.776506 8216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d780f2401c23420e90ff49507df4851a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b58adcd02d204af8810c43fa357d0ffa" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36095 } } peers { permanent_uuid: "324f3d9c6e264f09bd8ad5313baee734" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41813 } attrs { promote: false } } peers { permanent_uuid: "d780f2401c23420e90ff49507df4851a" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 42511 } attrs { promote: false } } }
I20260620 19:08:18.776600 8216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324f3d9c6e264f09bd8ad5313baee734 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:08:18.776685 8226 mvcc.cc:205] Tried to move back new op lower bound from 7299000314979508224 to 7299000314776731648. Current Snapshot: MvccSnapshot[applied={T|T < 7299000314979508224 or (T in {7299000314979508224})}]
I20260620 19:08:18.781642 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.778270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:41697 (local address 127.0.0.1:36022)
0620 19:08:18.778416 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.778427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:18.778481 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:18.778639 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:18.778640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:18.778642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:18.778706 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:18.778707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:18.779716 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:18.779716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:18.780823 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:18.780825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:18.781373 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:18.781374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:18.781448 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:18.781451 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:08:18.781529 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 19:08:18.781574 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":14}
I20260620 19:08:18.781687 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.778324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.0.203.3:41697)
0620 19:08:18.778484 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:18.778486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:18.778496 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:08:18.778529 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:18.778530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:18.778542 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:18.778770 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:18.778809 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:18.779609 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:18.780900 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:18.781274 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:18.781376 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:18.781390 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:18.781539 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 19:08:18.781606 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":57,"threads_started":1}
I20260620 19:08:18.782037 8081 tablet_service.cc:1467] Processing CreateTablet for tablet 9b23050642294c1d836a22bd2808713f (DEFAULT_TABLE table=test-table [id=3d01eb538dff482b89813fbaeddde453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:18.782202 8081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b23050642294c1d836a22bd2808713f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:18.782570 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.778323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:42711 (local address 127.0.0.1:50036)
0620 19:08:18.778441 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.778453 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:18.778506 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:18.778724 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:18.778725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:18.778726 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:18.778755 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:18.778756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:18.779712 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:18.779713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:18.781562 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:18.781564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:18.782320 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:18.782321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:18.782380 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:18.782384 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:08:18.782462 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 19:08:18.782500 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":1105}
I20260620 19:08:18.782573 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.778421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50036 (local address 127.0.203.2:42711)
0620 19:08:18.778603 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:18.778605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:18.778613 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:08:18.778636 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:18.778637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:18.778646 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:18.778814 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:18.778847 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:18.779616 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:18.781652 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:18.782224 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:18.782319 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:18.782330 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:18.782470 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 19:08:18.782507 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":51,"threads_started":1}
I20260620 19:08:18.783674 8235 tablet_bootstrap.cc:492] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16: Bootstrap starting.
I20260620 19:08:18.783737 7945 tablet_service.cc:1467] Processing CreateTablet for tablet 9b23050642294c1d836a22bd2808713f (DEFAULT_TABLE table=test-table [id=3d01eb538dff482b89813fbaeddde453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:18.783831 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.778347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:40925 (local address 127.0.0.1:33182)
0620 19:08:18.778693 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.778703 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:18.778761 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:18.778918 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:18.778919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:18.778921 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:18.778962 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:18.778963 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:18.779646 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:18.779647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:18.780822 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:18.780825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:18.783548 (+ 2723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:18.783549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:18.783636 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:18.783640 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:08:18.783706 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 19:08:18.783757 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":1,"thread_start_us":60,"threads_started":1}
I20260620 19:08:18.783856 7945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b23050642294c1d836a22bd2808713f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:18.784088 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.778526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33182 (local address 127.0.203.1:40925)
0620 19:08:18.778646 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:18.778648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:18.778814 (+ 166us) server_negotiation.cc:407] Connection header received
0620 19:08:18.778842 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:18.778843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:18.778855 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:18.778906 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:18.779012 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:18.779560 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:18.781425 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:18.783455 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:18.783537 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:18.783550 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:18.783912 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 19:08:18.784031 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
I20260620 19:08:18.784292 7808 tablet_service.cc:1467] Processing CreateTablet for tablet 9b23050642294c1d836a22bd2808713f (DEFAULT_TABLE table=test-table [id=3d01eb538dff482b89813fbaeddde453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:18.784407 7808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b23050642294c1d836a22bd2808713f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:18.785288 8236 tablet_bootstrap.cc:492] T 9b23050642294c1d836a22bd2808713f P 73d53fd8b46644c4a48293779879f814: Bootstrap starting.
I20260620 19:08:18.785651 8236 tablet_bootstrap.cc:654] T 9b23050642294c1d836a22bd2808713f P 73d53fd8b46644c4a48293779879f814: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:18.785764 8237 tablet_bootstrap.cc:492] T 9b23050642294c1d836a22bd2808713f P 6a241202a70f420ca87abfce500521fa: Bootstrap starting.
I20260620 19:08:18.785944 8235 tablet_bootstrap.cc:654] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:18.786172 8237 tablet_bootstrap.cc:654] T 9b23050642294c1d836a22bd2808713f P 6a241202a70f420ca87abfce500521fa: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:18.786643 8235 tablet_bootstrap.cc:492] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16: No bootstrap required, opened a new log
I20260620 19:08:18.786662 8236 tablet_bootstrap.cc:492] T 9b23050642294c1d836a22bd2808713f P 73d53fd8b46644c4a48293779879f814: No bootstrap required, opened a new log
I20260620 19:08:18.786681 8237 tablet_bootstrap.cc:492] T 9b23050642294c1d836a22bd2808713f P 6a241202a70f420ca87abfce500521fa: No bootstrap required, opened a new log
I20260620 19:08:18.786708 8236 ts_tablet_manager.cc:1397] T 9b23050642294c1d836a22bd2808713f P 73d53fd8b46644c4a48293779879f814: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:18.786710 8235 ts_tablet_manager.cc:1397] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 19:08:18.786721 8237 ts_tablet_manager.cc:1397] T 9b23050642294c1d836a22bd2808713f P 6a241202a70f420ca87abfce500521fa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:18.786825 8237 raft_consensus.cc:348] T 9b23050642294c1d836a22bd2808713f P 6a241202a70f420ca87abfce500521fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 } } peers { permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 } }
I20260620 19:08:18.786826 8236 raft_consensus.cc:348] T 9b23050642294c1d836a22bd2808713f P 73d53fd8b46644c4a48293779879f814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 } } peers { permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 } }
I20260620 19:08:18.786824 8235 raft_consensus.cc:348] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 } } peers { permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 } }
I20260620 19:08:18.786902 8237 raft_consensus.cc:374] T 9b23050642294c1d836a22bd2808713f P 6a241202a70f420ca87abfce500521fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:18.786902 8236 raft_consensus.cc:374] T 9b23050642294c1d836a22bd2808713f P 73d53fd8b46644c4a48293779879f814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:18.786919 8235 raft_consensus.cc:374] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:18.786926 8236 raft_consensus.cc:729] T 9b23050642294c1d836a22bd2808713f P 73d53fd8b46644c4a48293779879f814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d53fd8b46644c4a48293779879f814, State: Initialized, Role: FOLLOWER
I20260620 19:08:18.786926 8237 raft_consensus.cc:729] T 9b23050642294c1d836a22bd2808713f P 6a241202a70f420ca87abfce500521fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a241202a70f420ca87abfce500521fa, State: Initialized, Role: FOLLOWER
I20260620 19:08:18.786944 8235 raft_consensus.cc:729] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2d760a63644e13bde30b79f7913d16, State: Initialized, Role: FOLLOWER
I20260620 19:08:18.786993 8235 consensus_queue.cc:260] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 } } peers { permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 } }
I20260620 19:08:18.786993 8237 consensus_queue.cc:260] T 9b23050642294c1d836a22bd2808713f P 6a241202a70f420ca87abfce500521fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 } } peers { permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 } }
I20260620 19:08:18.787001 8236 consensus_queue.cc:260] T 9b23050642294c1d836a22bd2808713f P 73d53fd8b46644c4a48293779879f814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 } } peers { permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 } }
I20260620 19:08:18.787164 8237 ts_tablet_manager.cc:1428] T 9b23050642294c1d836a22bd2808713f P 6a241202a70f420ca87abfce500521fa: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:18.787184 8236 ts_tablet_manager.cc:1428] T 9b23050642294c1d836a22bd2808713f P 73d53fd8b46644c4a48293779879f814: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:18.787192 8235 ts_tablet_manager.cc:1428] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:18.787655 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.787444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.1:60837)
0620 19:08:18.787584 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.787625 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":84,"threads_started":1}
I20260620 19:08:18.788264 8147 heartbeater.cc:503] Master 127.0.203.60:42511 was elected leader, sending a full tablet report...
I20260620 19:08:18.788340 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.788128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.2:40595)
0620 19:08:18.788250 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.788289 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
I20260620 19:08:18.788620 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.787488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.3:56095)
0620 19:08:18.788548 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.788586 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":84,"threads_started":1}
I20260620 19:08:18.788808 7874 heartbeater.cc:503] Master 127.0.203.60:42511 was elected leader, sending a full tablet report...
I20260620 19:08:18.788822 8011 heartbeater.cc:503] Master 127.0.203.60:42511 was elected leader, sending a full tablet report...
I20260620 19:08:18.788985 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.788899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.2:42079)
0620 19:08:18.788935 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.788958 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 19:08:18.789027 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.788885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.3:36793)
0620 19:08:18.788967 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.789002 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 19:08:18.789593 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:18.789315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.1:44201)
0620 19:08:18.789385 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:18.789564 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 19:08:18.994235 8150 tablet.cc:2369] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:08:18.999909 8243 raft_consensus.cc:482] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:08:19.000020 8243 raft_consensus.cc:504] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 } } peers { permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 } }
I20260620 19:08:19.000379 8243 leader_election.cc:288] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a241202a70f420ca87abfce500521fa (127.0.203.1:40925), 73d53fd8b46644c4a48293779879f814 (127.0.203.2:42711)
I20260620 19:08:19.004432 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:19.000638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:35125 (local address 127.0.203.1:40925)
0620 19:08:19.000703 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:19.000704 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:19.000907 (+ 203us) server_negotiation.cc:407] Connection header received
0620 19:08:19.000994 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:19.000999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:19.001011 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:19.001075 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:19.001216 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:19.001920 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:19.003494 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:19.004048 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:19.004133 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:19.004146 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:19.004316 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 19:08:19.004361 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:08:19.004432 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:19.000636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:52359 (local address 127.0.203.2:42711)
0620 19:08:19.000686 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:19.000688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:19.000726 (+ 38us) server_negotiation.cc:407] Connection header received
0620 19:08:19.000829 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:19.000830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:19.000840 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:19.000917 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:19.001090 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:19.001858 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:19.003493 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:19.004047 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:19.004137 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:19.004147 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:19.004316 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 19:08:19.004361 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:08:19.004451 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:19.000567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:42711 (local address 127.0.203.3:52359)
0620 19:08:19.000648 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:19.000662 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:19.000739 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:19.000925 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:19.000926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:19.000929 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:19.000995 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:19.000997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:19.002000 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:19.002001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:19.003379 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:19.003382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:19.004140 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:19.004141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:19.004222 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:19.004226 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:08:19.004286 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 19:08:19.004345 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":81}
I20260620 19:08:19.004460 8247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:19.000573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:40925 (local address 127.0.203.3:35125)
0620 19:08:19.000822 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:19.000832 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:19.000922 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:19.001094 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:19.001095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:19.001097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:19.001146 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:19.001147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:19.002003 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:19.002003 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:19.003378 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:19.003382 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:19.004137 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:19.004138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:19.004222 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:19.004226 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:08:19.004286 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 19:08:19.004345 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":109,"thread_start_us":113,"threads_started":1}
I20260620 19:08:19.004798 7965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b23050642294c1d836a22bd2808713f" candidate_uuid: "ec2d760a63644e13bde30b79f7913d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d53fd8b46644c4a48293779879f814" is_pre_election: true
I20260620 19:08:19.004916 7965 raft_consensus.cc:2454] T 9b23050642294c1d836a22bd2808713f P 73d53fd8b46644c4a48293779879f814 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec2d760a63644e13bde30b79f7913d16 in term 0.
I20260620 19:08:19.005038 7828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b23050642294c1d836a22bd2808713f" candidate_uuid: "ec2d760a63644e13bde30b79f7913d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a241202a70f420ca87abfce500521fa" is_pre_election: true
I20260620 19:08:19.005116 8034 leader_election.cc:302] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73d53fd8b46644c4a48293779879f814, ec2d760a63644e13bde30b79f7913d16; no voters:
I20260620 19:08:19.005188 7828 raft_consensus.cc:2454] T 9b23050642294c1d836a22bd2808713f P 6a241202a70f420ca87abfce500521fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec2d760a63644e13bde30b79f7913d16 in term 0.
I20260620 19:08:19.005234 8243 raft_consensus.cc:2781] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:08:19.005277 8243 raft_consensus.cc:482] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:08:19.005308 8243 raft_consensus.cc:3037] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:19.005992 8243 raft_consensus.cc:504] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 } } peers { permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 } }
I20260620 19:08:19.006157 8243 leader_election.cc:288] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [CANDIDATE]: Term 1 election: Requested vote from peers 6a241202a70f420ca87abfce500521fa (127.0.203.1:40925), 73d53fd8b46644c4a48293779879f814 (127.0.203.2:42711)
I20260620 19:08:19.006295 7828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b23050642294c1d836a22bd2808713f" candidate_uuid: "ec2d760a63644e13bde30b79f7913d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a241202a70f420ca87abfce500521fa"
I20260620 19:08:19.006371 7828 raft_consensus.cc:3037] T 9b23050642294c1d836a22bd2808713f P 6a241202a70f420ca87abfce500521fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:19.006335 7965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b23050642294c1d836a22bd2808713f" candidate_uuid: "ec2d760a63644e13bde30b79f7913d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d53fd8b46644c4a48293779879f814"
I20260620 19:08:19.006441 7965 raft_consensus.cc:3037] T 9b23050642294c1d836a22bd2808713f P 73d53fd8b46644c4a48293779879f814 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:19.007169 7965 raft_consensus.cc:2454] T 9b23050642294c1d836a22bd2808713f P 73d53fd8b46644c4a48293779879f814 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec2d760a63644e13bde30b79f7913d16 in term 1.
I20260620 19:08:19.007203 7828 raft_consensus.cc:2454] T 9b23050642294c1d836a22bd2808713f P 6a241202a70f420ca87abfce500521fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec2d760a63644e13bde30b79f7913d16 in term 1.
I20260620 19:08:19.007320 8034 leader_election.cc:302] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73d53fd8b46644c4a48293779879f814, ec2d760a63644e13bde30b79f7913d16; no voters:
I20260620 19:08:19.007447 8243 raft_consensus.cc:2781] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:19.007654 8243 raft_consensus.cc:686] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [term 1 LEADER]: Becoming Leader. State: Replica: ec2d760a63644e13bde30b79f7913d16, State: Running, Role: LEADER
I20260620 19:08:19.007774 8243 consensus_queue.cc:237] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 } } peers { permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 } }
I20260620 19:08:19.007968 8247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:19.007871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.3:40201)
0620 19:08:19.007904 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:19.007940 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260620 19:08:19.008540 7694 catalog_manager.cc:5495] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec2d760a63644e13bde30b79f7913d16 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "ec2d760a63644e13bde30b79f7913d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec2d760a63644e13bde30b79f7913d16" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 } health_report { overall_health: UNKNOWN } } }
I20260620 19:08:19.064392 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:19.062010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36026 (local address 127.0.203.3:41697)
0620 19:08:19.062076 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:19.062078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:19.062087 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:08:19.062125 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:19.062126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:19.062150 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:19.062232 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:08:19.062345 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:19.062831 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:19.063622 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:19.063828 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:19.063901 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:19.064051 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:08:19.064157 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:08:19.064240 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:19.064282 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 19:08:19.064306 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:08:19.065511 7965 raft_consensus.cc:1261] T 9b23050642294c1d836a22bd2808713f P 73d53fd8b46644c4a48293779879f814 [term 1 FOLLOWER]: Refusing update from remote peer ec2d760a63644e13bde30b79f7913d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:19.065567 7828 raft_consensus.cc:1261] T 9b23050642294c1d836a22bd2808713f P 6a241202a70f420ca87abfce500521fa [term 1 FOLLOWER]: Refusing update from remote peer ec2d760a63644e13bde30b79f7913d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:19.065897 8248 consensus_queue.cc:1035] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a241202a70f420ca87abfce500521fa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:19.066025 8243 consensus_queue.cc:1035] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73d53fd8b46644c4a48293779879f814" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 42711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:19.066753 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:19.066642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.2:41195)
0620 19:08:19.066683 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:19.066717 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 19:08:19.067059 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:19.066958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.1:37789)
0620 19:08:19.067003 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:19.067027 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 19:08:19.068054 8247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:19.067940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.3:60009)
0620 19:08:19.067982 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:19.068008 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 19:08:19.068378 8247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:19.068298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.3:60057)
0620 19:08:19.068331 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:19.068353 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 19:08:19.068714 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:19.068617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.1:58787)
0620 19:08:19.068658 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:19.068685 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 19:08:19.070073 7695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:19.070165 7695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:19.070516 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:19.070432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36095 (local address 127.0.203.2:49885)
0620 19:08:19.070467 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:19.070490 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 19:08:19.071399 7695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d780f2401c23420e90ff49507df4851a: Sending DeleteTablet for 3 replicas of tablet 9b23050642294c1d836a22bd2808713f
I20260620 19:08:19.071702 7945 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b23050642294c1d836a22bd2808713f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:19 UTC) from {username='oryx', principal='oryx/127.0.203.60@KRBTEST.COM'} at 127.0.0.1:50036
I20260620 19:08:19.071944 8260 tablet_replica.cc:330] T 9b23050642294c1d836a22bd2808713f P 73d53fd8b46644c4a48293779879f814: stopping tablet replica
I20260620 19:08:19.071993 7808 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b23050642294c1d836a22bd2808713f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:19 UTC) from {username='oryx', principal='oryx/127.0.203.60@KRBTEST.COM'} at 127.0.0.1:33182
I20260620 19:08:19.072043 8081 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b23050642294c1d836a22bd2808713f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:19 UTC) from {username='oryx', principal='oryx/127.0.203.60@KRBTEST.COM'} at 127.0.0.1:36022
I20260620 19:08:19.072047 8260 raft_consensus.cc:2229] T 9b23050642294c1d836a22bd2808713f P 73d53fd8b46644c4a48293779879f814 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:19.072206 8262 tablet_replica.cc:330] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16: stopping tablet replica
I20260620 19:08:19.072260 8262 raft_consensus.cc:2229] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:08:19.072221 8260 raft_consensus.cc:2258] T 9b23050642294c1d836a22bd2808713f P 73d53fd8b46644c4a48293779879f814 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:19.072352 8262 raft_consensus.cc:2258] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:19.072412 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 7656
I20260620 19:08:19.072654 8262 ts_tablet_manager.cc:1905] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:19.074556 8262 ts_tablet_manager.cc:1918] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:19.074620 8262 log.cc:1196] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/9b23050642294c1d836a22bd2808713f
I20260620 19:08:19.074858 8262 ts_tablet_manager.cc:1939] T 9b23050642294c1d836a22bd2808713f P ec2d760a63644e13bde30b79f7913d16: Deleting consensus metadata
I20260620 19:08:19.075202 7680 catalog_manager.cc:4841] TS ec2d760a63644e13bde30b79f7913d16 (127.0.203.3:41697): tablet 9b23050642294c1d836a22bd2808713f (table test-table [id=3d01eb538dff482b89813fbaeddde453]) successfully deleted
I20260620 19:08:19.076218 8260 ts_tablet_manager.cc:1905] T 9b23050642294c1d836a22bd2808713f P 73d53fd8b46644c4a48293779879f814: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:19.078006 8260 ts_tablet_manager.cc:1918] T 9b23050642294c1d836a22bd2808713f P 73d53fd8b46644c4a48293779879f814: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:19.078078 8260 log.cc:1196] T 9b23050642294c1d836a22bd2808713f P 73d53fd8b46644c4a48293779879f814: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/9b23050642294c1d836a22bd2808713f
I20260620 19:08:19.078327 8260 ts_tablet_manager.cc:1939] T 9b23050642294c1d836a22bd2808713f P 73d53fd8b46644c4a48293779879f814: Deleting consensus metadata
I20260620 19:08:19.078613 7679 catalog_manager.cc:4841] TS 73d53fd8b46644c4a48293779879f814 (127.0.203.2:42711): tablet 9b23050642294c1d836a22bd2808713f (table test-table [id=3d01eb538dff482b89813fbaeddde453]) successfully deleted
I20260620 19:08:19.081027 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:19.080903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:40925 (local address 127.0.0.1:33190)
0620 19:08:19.080961 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:19.080993 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.1:40925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 19:08:19.081187 7680 catalog_manager.cc:4584] TS 6a241202a70f420ca87abfce500521fa (127.0.203.1:40925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b23050642294c1d836a22bd2808713f: Network error: Client connection negotiation failed: client connection to 127.0.203.1:40925: connect: Connection refused (error 111)
I20260620 19:08:19.081266 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 7882
I20260620 19:08:19.086596 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 8018
I20260620 19:08:19.091037 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 7375
I20260620 19:08:19.095822 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 7563
2026-06-20T19:08:19Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4823 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:08:19 dist-test-slave-wd30 krb5kdc[8265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:08:19 dist-test-slave-wd30 krb5kdc[8265](info): set up 2 sockets
Jun 20 19:08:19 dist-test-slave-wd30 krb5kdc[8265](info): commencing operation
krb5kdc: starting...
W20260620 19:08:20.127684 812 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:08:20 dist-test-slave-wd30 krb5kdc[8265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:08:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:08:20Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/oryx_127.0.203.62.keytab.
Entry for principal oryx/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/oryx_127.0.203.62.keytab.
I20260620 19:08:20.245682 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:36211
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/oryx_127.0.203.62.keytab
--principal=oryx/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:36211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:20.326983 8278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:20.327224 8278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:20.327247 8278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:20.328440 8278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:20.328480 8278 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 19:08:20.328495 8278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:20.328508 8278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:20.328562 8278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:20.328578 8278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:08:20.330109 8278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:36211
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/oryx_127.0.203.62.keytab
--principal=oryx/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:36211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:20.330365 8278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:20.330595 8278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:20.335433 8286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:20.335426 8284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:20.335513 8278 server_base.cc:1034] running on GCE node
W20260620 19:08:20.335427 8283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:20.335861 8278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:20.336175 8278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:20.337380 8278 hybrid_clock.cc:648] HybridClock initialized: now 1781982500337353 us; error 36 us; skew 500 ppm
Jun 20 19:08:20 dist-test-slave-wd30 krb5kdc[8265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982500, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:20.340286 8278 init.cc:377] Logged in from keytab as oryx/127.0.203.62@KRBTEST.COM (short username oryx)
I20260620 19:08:20.340746 8278 webserver.cc:460] Webserver started at http://127.0.203.62:40833/ using document root <none> and password file <none>
I20260620 19:08:20.340982 8278 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:20.341027 8278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:20.341177 8278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:20.342272 8278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "ed57b61f7c3d4b33b70d661eeee2f354"
format_stamp: "Formatted at 2026-06-20 19:08:20 on dist-test-slave-wd30"
server_key: "f8be00972a956492d08f4e7296fe184d"
server_key_iv: "1f3e38a51f31f2b06cc184896ef6f18b"
server_key_version: "encryptionkey@0"
I20260620 19:08:20.342671 8278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "ed57b61f7c3d4b33b70d661eeee2f354"
format_stamp: "Formatted at 2026-06-20 19:08:20 on dist-test-slave-wd30"
server_key: "f8be00972a956492d08f4e7296fe184d"
server_key_iv: "1f3e38a51f31f2b06cc184896ef6f18b"
server_key_version: "encryptionkey@0"
I20260620 19:08:20.344081 8278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:08:20.344795 8293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:20.344966 8278 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:08:20.345058 8278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "ed57b61f7c3d4b33b70d661eeee2f354"
format_stamp: "Formatted at 2026-06-20 19:08:20 on dist-test-slave-wd30"
server_key: "f8be00972a956492d08f4e7296fe184d"
server_key_iv: "1f3e38a51f31f2b06cc184896ef6f18b"
server_key_version: "encryptionkey@0"
I20260620 19:08:20.345139 8278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:20.359607 8278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:20.361420 8278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:20.361572 8278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:20.366255 8278 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:36211
I20260620 19:08:20.366302 8355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:36211 every 8 connection(s)
I20260620 19:08:20.366614 8278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:20.367240 8356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:20.369478 8356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354: Bootstrap starting.
I20260620 19:08:20.370065 8356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:20.370414 8356 log.cc:824] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:20.370468 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 8278
I20260620 19:08:20.370548 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:08:20.370735 812 external_mini_cluster.cc:1428] Setting key d2942abd00bf4eb8faa56458bcd43267
I20260620 19:08:20.371503 8356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354: No bootstrap required, opened a new log
I20260620 19:08:20.374085 8356 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed57b61f7c3d4b33b70d661eeee2f354" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36211 } }
I20260620 19:08:20.374253 8356 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:20.374289 8356 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed57b61f7c3d4b33b70d661eeee2f354, State: Initialized, Role: FOLLOWER
I20260620 19:08:20.374449 8356 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed57b61f7c3d4b33b70d661eeee2f354" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36211 } }
I20260620 19:08:20.374518 8356 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:20.374541 8356 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:20.374609 8356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:20.375218 8356 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed57b61f7c3d4b33b70d661eeee2f354" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36211 } }
I20260620 19:08:20.375317 8356 leader_election.cc:302] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed57b61f7c3d4b33b70d661eeee2f354; no voters:
I20260620 19:08:20.375470 8356 leader_election.cc:288] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:08:20.375522 8361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:20.375676 8361 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354 [term 1 LEADER]: Becoming Leader. State: Replica: ed57b61f7c3d4b33b70d661eeee2f354, State: Running, Role: LEADER
I20260620 19:08:20.375705 8356 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:20.375767 8361 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed57b61f7c3d4b33b70d661eeee2f354" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36211 } }
I20260620 19:08:20.376076 8362 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed57b61f7c3d4b33b70d661eeee2f354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed57b61f7c3d4b33b70d661eeee2f354" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36211 } } }
I20260620 19:08:20.376151 8362 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354 [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:20.376382 8363 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed57b61f7c3d4b33b70d661eeee2f354. Latest consensus state: current_term: 1 leader_uuid: "ed57b61f7c3d4b33b70d661eeee2f354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed57b61f7c3d4b33b70d661eeee2f354" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36211 } } }
I20260620 19:08:20.376662 8363 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354 [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:20.376732 8367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:08:20.377233 8367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 19:08:20 dist-test-slave-wd30 krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM
I20260620 19:08:20.378873 8367 catalog_manager.cc:1347] Generated new cluster ID: c2a9c75a3b6241d1a3d3abb9c2fbf8b1
I20260620 19:08:20.378922 8367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:08:20.382575 8360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:20.371239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.0.203.62:36211)
0620 19:08:20.371394 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:20.371404 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:20.371439 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:08:20.371985 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:20.371994 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:20.372058 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:20.372218 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:20.372800 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.374278 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.375104 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.375813 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.377353 (+ 1540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.378940 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:20.378943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:20.378944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:20.378954 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:20.380929 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:20.381351 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:20.381354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:20.381354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:20.381449 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:20.381660 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:20.381661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:20.381661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:20.381936 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:20.382071 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:20.382254 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 19:08:20.382378 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260620 19:08:20.414624 8367 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:08:20.415431 8367 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:20.420871 8367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354: Generated new TSK 0
I20260620 19:08:20.421068 8367 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/oryx_127.0.203.1.keytab.
Entry for principal oryx/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/oryx_127.0.203.1.keytab.
I20260620 19:08:20.445849 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36211
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/oryx_127.0.203.1.keytab
--principal=oryx/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:20.532423 8382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:20.532649 8382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:20.532689 8382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:20.533852 8382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:20.533918 8382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:20.533957 8382 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:08:20.534011 8382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:08:20.535514 8382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/oryx_127.0.203.1.keytab
--principal=oryx/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.203.62:36211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:20.535772 8382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:20.535993 8382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:20.541026 8390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:20.541024 8388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:20.541070 8382 server_base.cc:1034] running on GCE node
W20260620 19:08:20.541024 8387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:20.541461 8382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:20.541741 8382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:20.543081 8382 hybrid_clock.cc:648] HybridClock initialized: now 1781982500543077 us; error 35 us; skew 500 ppm
Jun 20 19:08:20 dist-test-slave-wd30 krb5kdc[8265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982500, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:20.548969 8382 init.cc:377] Logged in from keytab as oryx/127.0.203.1@KRBTEST.COM (short username oryx)
I20260620 19:08:20.549437 8382 webserver.cc:460] Webserver started at http://127.0.203.1:41843/ using document root <none> and password file <none>
I20260620 19:08:20.549665 8382 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:20.549736 8382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:20.549890 8382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:20.550952 8382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "b869b8b864be40fa9446f0e98683c84f"
format_stamp: "Formatted at 2026-06-20 19:08:20 on dist-test-slave-wd30"
server_key: "12317ee8a0b9c0a8db6dc9ae31ea8562"
server_key_iv: "fe7bde4e638bd75822db06f830906b65"
server_key_version: "encryptionkey@0"
I20260620 19:08:20.551342 8382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "b869b8b864be40fa9446f0e98683c84f"
format_stamp: "Formatted at 2026-06-20 19:08:20 on dist-test-slave-wd30"
server_key: "12317ee8a0b9c0a8db6dc9ae31ea8562"
server_key_iv: "fe7bde4e638bd75822db06f830906b65"
server_key_version: "encryptionkey@0"
I20260620 19:08:20.552641 8382 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:08:20.553352 8397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:20.553516 8382 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:20.553615 8382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "b869b8b864be40fa9446f0e98683c84f"
format_stamp: "Formatted at 2026-06-20 19:08:20 on dist-test-slave-wd30"
server_key: "12317ee8a0b9c0a8db6dc9ae31ea8562"
server_key_iv: "fe7bde4e638bd75822db06f830906b65"
server_key_version: "encryptionkey@0"
I20260620 19:08:20.553696 8382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:20.569705 8382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:20.572088 8382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:20.572255 8382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:20.572796 8382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:20.572866 8382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:20.572903 8382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:20.572952 8382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:08:20 dist-test-slave-wd30 krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982500, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM
I20260620 19:08:20.580175 8382 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:37653
I20260620 19:08:20.580327 8512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:37653 every 8 connection(s)
I20260620 19:08:20.580607 8382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:08:20.581207 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 8382
I20260620 19:08:20.581290 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:08:20.581453 812 external_mini_cluster.cc:1428] Setting key 381b54c28a93ea82f147e3841bc0af48
I20260620 19:08:20.583551 8360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:20.573293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:43387 (local address 127.0.203.62:36211)
0620 19:08:20.573372 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:20.573375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:20.573552 (+ 177us) server_negotiation.cc:407] Connection header received
0620 19:08:20.574024 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:20.574026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:20.574040 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:20.574124 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:20.574952 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.575865 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.577147 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.577545 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.577626 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.580026 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:20.580027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:20.580028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:20.580035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:20.582145 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:20.582549 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:20.582551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:20.582551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:20.582615 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:20.582870 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:20.582871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:20.582872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:20.582991 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:20.583070 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:20.583374 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 19:08:20.583457 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:08:20.583748 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:20.573268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36211 (local address 127.0.203.1:43387)
0620 19:08:20.573411 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:20.573451 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:20.573930 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:20.574177 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:20.574180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:20.574536 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:20.574834 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:20.574842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.575964 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.575965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:20.577047 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:20.577049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.577650 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.577651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:20.577789 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.578298 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:20.578306 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:20.579905 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:20.582231 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:20.582237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:20.582240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:20.582462 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:20.582689 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:20.582690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:20.582690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:20.582793 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:20.583062 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:20.583068 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:20.583237 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 19:08:20.583353 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 19:08:20.583513 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260620 19:08:20.584359 8513 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36211
I20260620 19:08:20.584458 8513 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:20.584635 8513 heartbeater.cc:511] Master 127.0.203.62:36211 requested a full tablet report, sending...
I20260620 19:08:20.585628 8310 ts_manager.cc:194] Registered new tserver with Master: b869b8b864be40fa9446f0e98683c84f (127.0.203.1:37653)
I20260620 19:08:20.586593 8310 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:43387
WARNING: no policy specified for oryx/127.0.203.2@KRBTEST.COM; defaulting to no policy
I20260620 19:08:20.591269 8360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:20.585546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:51047 (local address 127.0.203.62:36211)
0620 19:08:20.585581 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:20.585583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:20.585622 (+ 39us) server_negotiation.cc:407] Connection header received
0620 19:08:20.585691 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:20.585691 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:20.585701 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:20.585742 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:20.586080 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.586543 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.587456 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.587652 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.587737 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.588631 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:20.588632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:20.588633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:20.588639 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:20.590081 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:20.590416 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:20.590417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:20.590418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:20.590471 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:20.590722 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:20.590722 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:20.590723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:20.590836 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:20.590911 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:20.591086 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 19:08:20.591181 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:08:20.591282 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:20.585402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36211 (local address 127.0.203.1:51047)
0620 19:08:20.585517 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:20.585579 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:20.585638 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:20.585778 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:20.585779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:20.585933 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:20.585993 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:20.585994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.586635 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.586636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:20.587376 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:20.587378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.587737 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.587738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:20.587802 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.588225 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:20.588230 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:20.588526 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:20.590165 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:20.590166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:20.590166 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:20.590343 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:20.590586 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:20.590587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:20.590588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:20.590645 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:20.590903 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:20.590904 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:20.590946 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 19:08:20.591044 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 19:08:20.591176 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/oryx_127.0.203.2.keytab.
Entry for principal oryx/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/oryx_127.0.203.2.keytab.
I20260620 19:08:20.605898 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36211
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/oryx_127.0.203.2.keytab
--principal=oryx/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:20.687386 8522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:20.687608 8522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:20.687659 8522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:20.688856 8522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:20.688925 8522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:20.688961 8522 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:08:20.689018 8522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:08:20.690488 8522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/oryx_127.0.203.2.keytab
--principal=oryx/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.203.62:36211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:20.690761 8522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:20.690987 8522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:20.695909 8528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:20.695895 8530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:20.695888 8527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:20.696175 8522 server_base.cc:1034] running on GCE node
I20260620 19:08:20.696445 8522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:20.696727 8522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:20.697885 8522 hybrid_clock.cc:648] HybridClock initialized: now 1781982500697862 us; error 40 us; skew 500 ppm
Jun 20 19:08:20 dist-test-slave-wd30 krb5kdc[8265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982500, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:20.700361 8522 init.cc:377] Logged in from keytab as oryx/127.0.203.2@KRBTEST.COM (short username oryx)
I20260620 19:08:20.700848 8522 webserver.cc:460] Webserver started at http://127.0.203.2:42051/ using document root <none> and password file <none>
I20260620 19:08:20.701098 8522 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:20.701166 8522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:20.701324 8522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:20.702354 8522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696"
format_stamp: "Formatted at 2026-06-20 19:08:20 on dist-test-slave-wd30"
server_key: "3bab720524d7ba761a583a713160277b"
server_key_iv: "9c88d648088fc50ef76a63ddabc5c347"
server_key_version: "encryptionkey@0"
I20260620 19:08:20.702713 8522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696"
format_stamp: "Formatted at 2026-06-20 19:08:20 on dist-test-slave-wd30"
server_key: "3bab720524d7ba761a583a713160277b"
server_key_iv: "9c88d648088fc50ef76a63ddabc5c347"
server_key_version: "encryptionkey@0"
I20260620 19:08:20.704061 8522 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:08:20.704758 8537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:20.704922 8522 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:20.705026 8522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696"
format_stamp: "Formatted at 2026-06-20 19:08:20 on dist-test-slave-wd30"
server_key: "3bab720524d7ba761a583a713160277b"
server_key_iv: "9c88d648088fc50ef76a63ddabc5c347"
server_key_version: "encryptionkey@0"
I20260620 19:08:20.705098 8522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:20.718205 8522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:20.720034 8522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:20.720194 8522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:20.720717 8522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:20.720783 8522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:20.720825 8522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:20.720853 8522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:08:20 dist-test-slave-wd30 krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982500, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM
I20260620 19:08:20.728226 8522 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:40787
I20260620 19:08:20.728632 8522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:08:20.728654 8652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:40787 every 8 connection(s)
I20260620 19:08:20.730252 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 8522
I20260620 19:08:20.730358 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:08:20.730573 812 external_mini_cluster.cc:1428] Setting key 1181582f0efd905c3072105b1b4a0d51
I20260620 19:08:20.730690 8360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:20.721086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:50731 (local address 127.0.203.62:36211)
0620 19:08:20.721140 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:20.721142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:20.721405 (+ 263us) server_negotiation.cc:407] Connection header received
0620 19:08:20.722111 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:20.722112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:20.722122 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:20.722186 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:20.723138 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.723655 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.724687 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.724892 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.724970 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.727833 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:20.727834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:20.727835 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:20.727843 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:20.729396 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:20.729739 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:20.729739 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:20.729740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:20.729792 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:20.729999 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:20.729999 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:20.730000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:20.730099 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:20.730153 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:20.730516 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 19:08:20.730579 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:08:20.730886 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:20.721067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36211 (local address 127.0.203.2:50731)
0620 19:08:20.721246 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:20.721298 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:20.722005 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:20.722288 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:20.722293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:20.722699 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:20.723014 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:20.723022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.723757 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.723758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:20.724579 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:20.724582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.724975 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.724977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:20.725100 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.725594 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:20.725603 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:20.727731 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:20.729466 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:20.729470 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:20.729471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:20.729670 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:20.729850 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:20.729851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:20.729851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:20.729945 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:20.730168 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:20.730174 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:20.730384 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 19:08:20.730504 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 19:08:20.730672 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260620 19:08:20.731525 8653 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36211
I20260620 19:08:20.731618 8653 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:20.731808 8653 heartbeater.cc:511] Master 127.0.203.62:36211 requested a full tablet report, sending...
I20260620 19:08:20.732307 8309 ts_manager.cc:194] Registered new tserver with Master: 2073bd7db4d14d9b8bd9ea0c5cb65696 (127.0.203.2:40787)
I20260620 19:08:20.732851 8309 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:50731
I20260620 19:08:20.738848 8360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:20.732594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:45175 (local address 127.0.203.62:36211)
0620 19:08:20.732632 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:20.732633 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:20.732750 (+ 117us) server_negotiation.cc:407] Connection header received
0620 19:08:20.732863 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:20.732864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:20.732874 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:20.732905 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:20.733396 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.733941 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.735032 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.735274 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.735327 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.736321 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:20.736322 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:20.736323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:20.736328 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:20.737761 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:20.738138 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:20.738138 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:20.738139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:20.738185 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:20.738385 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:20.738385 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:20.738386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:20.738507 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:20.738555 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:20.738732 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 19:08:20.738789 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:08:20.738967 8659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:20.732547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36211 (local address 127.0.203.2:45175)
0620 19:08:20.732687 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:20.732698 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:20.732775 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:20.733027 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:20.733028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:20.733234 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:20.733313 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:20.733315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.734033 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.734034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:20.734950 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:20.734952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.735354 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.735355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:20.735439 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.735825 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:20.735830 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:20.736227 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:20.737847 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:20.737848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:20.737848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:20.738061 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:20.738250 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:20.738250 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:20.738251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:20.738318 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:20.738575 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:20.738576 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:20.738629 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 19:08:20.738714 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 19:08:20.738825 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for oryx/127.0.203.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/oryx_127.0.203.3.keytab.
Entry for principal oryx/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/oryx_127.0.203.3.keytab.
I20260620 19:08:20.758015 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36211
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/oryx_127.0.203.3.keytab
--principal=oryx/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:20.839457 8662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:20.839650 8662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:20.839674 8662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:20.840838 8662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:20.840881 8662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:20.840902 8662 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:08:20.840919 8662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:08:20.842461 8662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/krb5kdc/oryx_127.0.203.3.keytab
--principal=oryx/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.203.62:36211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:20.842710 8662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:20.842952 8662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:20.848054 8670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:20.848079 8668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:20.848130 8662 server_base.cc:1034] running on GCE node
W20260620 19:08:20.848086 8667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:20.848505 8662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:20.848816 8662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:20.849980 8662 hybrid_clock.cc:648] HybridClock initialized: now 1781982500849965 us; error 27 us; skew 500 ppm
Jun 20 19:08:20 dist-test-slave-wd30 krb5kdc[8265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982500, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:20.852458 8662 init.cc:377] Logged in from keytab as oryx/127.0.203.3@KRBTEST.COM (short username oryx)
I20260620 19:08:20.852922 8662 webserver.cc:460] Webserver started at http://127.0.203.3:36959/ using document root <none> and password file <none>
I20260620 19:08:20.853163 8662 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:20.853230 8662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:20.853390 8662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:20.854425 8662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "8b9ddaf367644bacb3aac674f33582c1"
format_stamp: "Formatted at 2026-06-20 19:08:20 on dist-test-slave-wd30"
server_key: "7fb20c5ce77226a0d84bbfc4e00aa319"
server_key_iv: "9f2e76044d5b8a20ff2a3e93240ec74c"
server_key_version: "encryptionkey@0"
I20260620 19:08:20.854794 8662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "8b9ddaf367644bacb3aac674f33582c1"
format_stamp: "Formatted at 2026-06-20 19:08:20 on dist-test-slave-wd30"
server_key: "7fb20c5ce77226a0d84bbfc4e00aa319"
server_key_iv: "9f2e76044d5b8a20ff2a3e93240ec74c"
server_key_version: "encryptionkey@0"
I20260620 19:08:20.856244 8662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:08:20.857050 8677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:20.857228 8662 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:20.857342 8662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "8b9ddaf367644bacb3aac674f33582c1"
format_stamp: "Formatted at 2026-06-20 19:08:20 on dist-test-slave-wd30"
server_key: "7fb20c5ce77226a0d84bbfc4e00aa319"
server_key_iv: "9f2e76044d5b8a20ff2a3e93240ec74c"
server_key_version: "encryptionkey@0"
I20260620 19:08:20.857415 8662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:20.873754 8662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:20.875982 8662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:20.876158 8662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:20.876827 8662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:20.876883 8662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:20.876931 8662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:20.876962 8662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:08:20 dist-test-slave-wd30 krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982500, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM
I20260620 19:08:20.884589 8662 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:35531
I20260620 19:08:20.884684 8792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:35531 every 8 connection(s)
I20260620 19:08:20.885021 8662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:08:20.887970 8360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:20.877236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:57007 (local address 127.0.203.62:36211)
0620 19:08:20.877285 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:20.877287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:20.877501 (+ 214us) server_negotiation.cc:407] Connection header received
0620 19:08:20.878072 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:20.878074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:20.878090 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:20.878198 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:20.879060 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.879919 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.881657 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.881991 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.882067 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.885022 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:20.885024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:20.885024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:20.885033 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:20.886541 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:20.886947 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:20.886948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:20.886949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:20.887006 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:20.887311 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:20.887312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:20.887312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:20.887439 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:20.887523 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:20.887789 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 19:08:20.887871 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:08:20.888166 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:20.877220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36211 (local address 127.0.203.3:57007)
0620 19:08:20.877366 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:20.877407 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:20.877968 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:20.878284 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:20.878293 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:20.878656 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:20.878954 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:20.878959 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.880059 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.880060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:20.881532 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:20.881536 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.882110 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.882111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:20.882283 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.882965 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:20.882976 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:20.884924 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:20.886634 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:20.886641 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:20.886643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:20.886869 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:20.887083 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:20.887083 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:20.887084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:20.887226 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:20.887513 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:20.887519 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:20.887659 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 19:08:20.887767 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 19:08:20.887928 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":83,"threads_started":1}
I20260620 19:08:20.888669 8793 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36211
I20260620 19:08:20.888756 8793 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:20.888950 8793 heartbeater.cc:511] Master 127.0.203.62:36211 requested a full tablet report, sending...
I20260620 19:08:20.889387 8310 ts_manager.cc:194] Registered new tserver with Master: 8b9ddaf367644bacb3aac674f33582c1 (127.0.203.3:35531)
I20260620 19:08:20.889917 8310 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:57007
I20260620 19:08:20.892935 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 8662
I20260620 19:08:20.893041 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:08:20.893249 812 external_mini_cluster.cc:1428] Setting key 55982676cd580c8af26195eeca208933
I20260620 19:08:20.894017 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:08:20.895515 8360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:20.889820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:44615 (local address 127.0.203.62:36211)
0620 19:08:20.890056 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:20.890059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:20.890067 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:08:20.890086 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:20.890086 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:20.890097 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:20.890150 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:20.890508 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.890982 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.891744 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.891950 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.892032 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.892875 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:20.892877 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:20.892877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:20.892884 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:20.894376 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:20.894716 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:20.894717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:20.894718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:20.894775 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:20.894975 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:20.894976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:20.894977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:20.895086 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:20.895180 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:20.895355 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 19:08:20.895425 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260620 19:08:20.895529 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:20.889748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36211 (local address 127.0.203.3:44615)
0620 19:08:20.889866 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:20.889877 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:20.889930 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:20.890165 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:20.890166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:20.890372 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:20.890427 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:20.890429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.891075 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.891076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:20.891639 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:20.891641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.892035 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.892036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:20.892099 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.892478 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:20.892483 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:20.892766 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:20.894455 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:20.894456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:20.894456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:20.894630 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:20.894845 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:20.894846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:20.894846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:20.894911 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:20.895170 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:20.895171 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:20.895216 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 19:08:20.895323 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 19:08:20.895427 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
Jun 20 19:08:20 dist-test-slave-wd30 krb5kdc[8265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:08:20 dist-test-slave-wd30 krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:20 dist-test-slave-wd30 krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:08:20.914947 8360 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:20.909825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.0.203.62:36211)
0620 19:08:20.909887 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:20.909889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:20.909901 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:08:20.909935 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:20.909936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:20.909945 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:20.909995 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:20.910519 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.911012 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.911795 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.911981 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.912028 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.914847 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38506: BlockingRecv error: recv got EOF from 127.0.0.1:38506 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 19:08:20 dist-test-slave-wd30 krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM
I20260620 19:08:20.922210 8360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:20.915676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38512 (local address 127.0.203.62:36211)
0620 19:08:20.915726 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:20.915728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:20.915797 (+ 69us) server_negotiation.cc:407] Connection header received
0620 19:08:20.915832 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:20.915832 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:20.915840 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:20.915889 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:20.916313 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.916764 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.917374 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.917541 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.917589 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.919685 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:20.919687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:20.919687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:20.919692 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:20.921102 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:20.921424 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:20.921425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:20.921426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:20.921483 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:20.921684 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:20.921685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:20.921686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:20.921804 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:20.921907 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:20.922031 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 19:08:20.922108 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 19:08:20.923785 8310 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:08:20.925336 8310 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:08:20.931931 8815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:20.927542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:37653 (local address 127.0.0.1:37306)
0620 19:08:20.927782 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:20.927791 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:20.927832 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:20.928448 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:20.928449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:20.928451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:20.928497 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:20.928498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.929734 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.929735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:20.930698 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:20.930700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.931623 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.931624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:20.931700 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.931715 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 19:08:20.931792 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 19:08:20.931843 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":79,"threads_started":1}
I20260620 19:08:20.931962 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:20.927640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37306 (local address 127.0.203.1:37653)
0620 19:08:20.928151 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:20.928155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:20.928183 (+ 28us) server_negotiation.cc:407] Connection header received
0620 19:08:20.928222 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:20.928224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:20.928239 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:20.928321 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:20.928579 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.929651 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.930790 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.931544 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.931620 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.931699 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:20.931821 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 19:08:20.931858 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":69,"threads_started":1}
I20260620 19:08:20.932372 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:20.927758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46452 (local address 127.0.203.3:35531)
0620 19:08:20.928440 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:20.928445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:20.928454 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:08:20.928490 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:20.928492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:20.928507 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:20.928579 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:20.928759 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.929450 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.931196 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.931846 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.931906 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.931978 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:20.932187 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 19:08:20.932261 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":81,"threads_started":1}
I20260620 19:08:20.932389 8816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:20.927535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:35531 (local address 127.0.0.1:46452)
0620 19:08:20.927698 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:20.927739 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:20.927822 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:20.928647 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:20.928648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:20.928650 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:20.928687 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:20.928688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.929554 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.929555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:20.930890 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:20.930891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.932013 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.932013 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:20.932073 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.932077 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:08:20.932153 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:08:20.932196 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":36,"thread_start_us":51,"threads_started":1}
I20260620 19:08:20.932390 8820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:20.927758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.0.203.2:40787)
0620 19:08:20.927944 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:20.927951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:20.928071 (+ 120us) server_negotiation.cc:407] Connection header received
0620 19:08:20.928109 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:20.928111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:20.928124 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:20.928186 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:20.928367 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.929568 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.930976 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.931731 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.931808 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.931876 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:20.932242 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 19:08:20.932288 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":98,"threads_started":1}
I20260620 19:08:20.932696 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:20.927563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:40787 (local address 127.0.0.1:60452)
0620 19:08:20.927994 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:20.928002 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:20.928054 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:20.928220 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:20.928224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:20.928225 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:20.928292 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:20.928293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.929634 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.929635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:20.930888 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:20.930891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:20.931911 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:20.931912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:20.931989 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:20.931993 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:08:20.932129 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 19:08:20.932173 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":66,"thread_start_us":75,"threads_started":1}
I20260620 19:08:20.933008 8727 tablet_service.cc:1467] Processing CreateTablet for tablet 514eebdf8d6640be835eb36172df00e2 (DEFAULT_TABLE table=test-table [id=fb5d899cd43d4581bb4db6b77670d31e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:20.933007 8447 tablet_service.cc:1467] Processing CreateTablet for tablet 514eebdf8d6640be835eb36172df00e2 (DEFAULT_TABLE table=test-table [id=fb5d899cd43d4581bb4db6b77670d31e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:20.933255 8587 tablet_service.cc:1467] Processing CreateTablet for tablet 514eebdf8d6640be835eb36172df00e2 (DEFAULT_TABLE table=test-table [id=fb5d899cd43d4581bb4db6b77670d31e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:20.933358 8447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514eebdf8d6640be835eb36172df00e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:20.933370 8727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514eebdf8d6640be835eb36172df00e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:20.933539 8587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514eebdf8d6640be835eb36172df00e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:20.936635 8821 tablet_bootstrap.cc:492] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f: Bootstrap starting.
I20260620 19:08:20.937075 8822 tablet_bootstrap.cc:492] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1: Bootstrap starting.
I20260620 19:08:20.937384 8821 tablet_bootstrap.cc:654] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:20.937711 8822 tablet_bootstrap.cc:654] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:20.937711 8821 log.cc:824] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:20.938026 8822 log.cc:824] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:20.938333 8823 tablet_bootstrap.cc:492] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Bootstrap starting.
I20260620 19:08:20.938519 8821 tablet_bootstrap.cc:492] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f: No bootstrap required, opened a new log
I20260620 19:08:20.938604 8821 ts_tablet_manager.cc:1397] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:08:20.938767 8822 tablet_bootstrap.cc:492] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1: No bootstrap required, opened a new log
I20260620 19:08:20.938843 8822 ts_tablet_manager.cc:1397] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:08:20.938956 8823 tablet_bootstrap.cc:654] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:20.939363 8823 log.cc:824] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:20.940104 8823 tablet_bootstrap.cc:492] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696: No bootstrap required, opened a new log
I20260620 19:08:20.940174 8823 ts_tablet_manager.cc:1397] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:08:20.940647 8822 raft_consensus.cc:348] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } } peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } }
I20260620 19:08:20.940802 8822 raft_consensus.cc:374] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:20.940845 8822 raft_consensus.cc:729] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9ddaf367644bacb3aac674f33582c1, State: Initialized, Role: FOLLOWER
I20260620 19:08:20.940971 8822 consensus_queue.cc:260] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } } peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } }
I20260620 19:08:20.940943 8821 raft_consensus.cc:348] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } } peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } }
I20260620 19:08:20.941097 8821 raft_consensus.cc:374] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:20.941130 8821 raft_consensus.cc:729] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b869b8b864be40fa9446f0e98683c84f, State: Initialized, Role: FOLLOWER
I20260620 19:08:20.941241 8822 ts_tablet_manager.cc:1428] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 19:08:20.941239 8821 consensus_queue.cc:260] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } } peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } }
I20260620 19:08:20.941356 8793 heartbeater.cc:503] Master 127.0.203.62:36211 was elected leader, sending a full tablet report...
I20260620 19:08:20.941569 8513 heartbeater.cc:503] Master 127.0.203.62:36211 was elected leader, sending a full tablet report...
I20260620 19:08:20.941509 8821 ts_tablet_manager.cc:1428] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:08:20.942157 8823 raft_consensus.cc:348] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } } peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } }
I20260620 19:08:20.942335 8823 raft_consensus.cc:374] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:20.942384 8823 raft_consensus.cc:729] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2073bd7db4d14d9b8bd9ea0c5cb65696, State: Initialized, Role: FOLLOWER
I20260620 19:08:20.942484 8823 consensus_queue.cc:260] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } } peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } }
I20260620 19:08:20.942778 8823 ts_tablet_manager.cc:1428] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 19:08:20.942799 8653 heartbeater.cc:503] Master 127.0.203.62:36211 was elected leader, sending a full tablet report...
W20260620 19:08:20.979218 8654 tablet.cc:2369] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:08:21.081619 8514 tablet.cc:2369] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:08:21.135598 8794 tablet.cc:2369] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:08:21.288321 8828 raft_consensus.cc:482] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:08:21.288487 8828 raft_consensus.cc:504] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } } peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } }
I20260620 19:08:21.288894 8828 leader_election.cc:288] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b9ddaf367644bacb3aac674f33582c1 (127.0.203.3:35531), 2073bd7db4d14d9b8bd9ea0c5cb65696 (127.0.203.2:40787)
I20260620 19:08:21.291929 8820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.289166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:57297 (local address 127.0.203.2:40787)
0620 19:08:21.289221 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:21.289223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:21.289295 (+ 72us) server_negotiation.cc:407] Connection header received
0620 19:08:21.289443 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:21.289445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:21.289459 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:21.289515 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:21.289696 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.290194 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.291187 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.291555 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.291608 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.291616 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:21.291807 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 19:08:21.291854 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 19:08:21.291973 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.289302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:40359 (local address 127.0.203.3:35531)
0620 19:08:21.289345 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:21.289347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:21.289358 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:08:21.289504 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:21.289505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:21.289518 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:21.289575 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:21.289774 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.290210 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.291247 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.291621 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.291689 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.291699 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:21.291871 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 19:08:21.291910 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:08:21.291953 8830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.289085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:40787 (local address 127.0.203.1:57297)
0620 19:08:21.289232 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:21.289244 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:21.289308 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:21.289527 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:21.289528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:21.289531 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:21.289618 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.289620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.290285 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.290286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.291093 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.291095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.291631 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.291632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.291708 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.291713 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:08:21.291767 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 19:08:21.291806 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":1,"thread_start_us":71,"threads_started":1}
I20260620 19:08:21.292222 8831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.289166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:35531 (local address 127.0.203.1:40359)
0620 19:08:21.289308 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:21.289317 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:21.289400 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:21.289601 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:21.289602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:21.289604 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:21.289685 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.289687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.290325 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.290326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.291105 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.291107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.291705 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.291706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.291785 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.291789 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:08:21.291849 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 19:08:21.291885 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":60,"thread_start_us":75,"threads_started":1}
I20260620 19:08:21.292307 8606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514eebdf8d6640be835eb36172df00e2" candidate_uuid: "b869b8b864be40fa9446f0e98683c84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" is_pre_election: true
I20260620 19:08:21.292485 8606 raft_consensus.cc:2454] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b869b8b864be40fa9446f0e98683c84f in term 0.
I20260620 19:08:21.292562 8747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514eebdf8d6640be835eb36172df00e2" candidate_uuid: "b869b8b864be40fa9446f0e98683c84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9ddaf367644bacb3aac674f33582c1" is_pre_election: true
I20260620 19:08:21.292718 8747 raft_consensus.cc:2454] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b869b8b864be40fa9446f0e98683c84f in term 0.
I20260620 19:08:21.292735 8400 leader_election.cc:302] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2073bd7db4d14d9b8bd9ea0c5cb65696, b869b8b864be40fa9446f0e98683c84f; no voters:
I20260620 19:08:21.292985 8828 raft_consensus.cc:2781] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:08:21.293036 8828 raft_consensus.cc:482] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:08:21.293052 8828 raft_consensus.cc:3037] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:21.293756 8828 raft_consensus.cc:504] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } } peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } }
I20260620 19:08:21.293908 8828 leader_election.cc:288] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [CANDIDATE]: Term 1 election: Requested vote from peers 8b9ddaf367644bacb3aac674f33582c1 (127.0.203.3:35531), 2073bd7db4d14d9b8bd9ea0c5cb65696 (127.0.203.2:40787)
I20260620 19:08:21.294066 8747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514eebdf8d6640be835eb36172df00e2" candidate_uuid: "b869b8b864be40fa9446f0e98683c84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9ddaf367644bacb3aac674f33582c1"
I20260620 19:08:21.294067 8606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514eebdf8d6640be835eb36172df00e2" candidate_uuid: "b869b8b864be40fa9446f0e98683c84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696"
I20260620 19:08:21.294143 8747 raft_consensus.cc:3037] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:21.294143 8606 raft_consensus.cc:3037] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:21.294956 8606 raft_consensus.cc:2454] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b869b8b864be40fa9446f0e98683c84f in term 1.
I20260620 19:08:21.295017 8747 raft_consensus.cc:2454] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b869b8b864be40fa9446f0e98683c84f in term 1.
I20260620 19:08:21.295142 8400 leader_election.cc:302] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2073bd7db4d14d9b8bd9ea0c5cb65696, b869b8b864be40fa9446f0e98683c84f; no voters:
I20260620 19:08:21.295251 8828 raft_consensus.cc:2781] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:21.295385 8828 raft_consensus.cc:686] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [term 1 LEADER]: Becoming Leader. State: Replica: b869b8b864be40fa9446f0e98683c84f, State: Running, Role: LEADER
I20260620 19:08:21.295480 8828 consensus_queue.cc:237] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } } peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } }
I20260620 19:08:21.296293 8309 catalog_manager.cc:5495] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f reported cstate change: term changed from 0 to 1, leader changed from <none> to b869b8b864be40fa9446f0e98683c84f (127.0.203.1). New cstate: current_term: 1 leader_uuid: "b869b8b864be40fa9446f0e98683c84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20260620 19:08:21.371311 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.368874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.0.203.1:37653)
0620 19:08:21.368944 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:21.368945 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:21.368953 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:08:21.368986 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:21.368987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:21.369006 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:21.369082 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:08:21.369229 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.369713 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.370500 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.370705 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.370789 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.370973 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:08:21.371077 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:08:21.371193 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:21.371210 (+ 17us) server_negotiation.cc:299] Negotiation successful
0620 19:08:21.371230 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:08:21.373139 8606 raft_consensus.cc:1261] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 1 FOLLOWER]: Refusing update from remote peer b869b8b864be40fa9446f0e98683c84f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:21.373139 8747 raft_consensus.cc:1261] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1 [term 1 FOLLOWER]: Refusing update from remote peer b869b8b864be40fa9446f0e98683c84f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:21.373514 8832 consensus_queue.cc:1035] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:21.373656 8828 consensus_queue.cc:1035] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:21.376154 8837 mvcc.cc:205] Tried to move back new op lower bound from 7299000325621297152 to 7299000325306761216. Current Snapshot: MvccSnapshot[applied={T|T < 7299000325621297152}]
I20260620 19:08:21.376246 8836 mvcc.cc:205] Tried to move back new op lower bound from 7299000325621297152 to 7299000325306761216. Current Snapshot: MvccSnapshot[applied={T|T < 7299000325621297152}]
I20260620 19:08:21.378376 8310 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:21.378461 8310 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:21.379222 8310 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354: Sending DeleteTablet for 3 replicas of tablet 514eebdf8d6640be835eb36172df00e2
I20260620 19:08:21.379547 8447 tablet_service.cc:1514] Processing DeleteTablet for tablet 514eebdf8d6640be835eb36172df00e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:21 UTC) from {username='oryx', principal='oryx/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:37306
I20260620 19:08:21.379550 8587 tablet_service.cc:1514] Processing DeleteTablet for tablet 514eebdf8d6640be835eb36172df00e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:21 UTC) from {username='oryx', principal='oryx/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:60452
I20260620 19:08:21.379559 8727 tablet_service.cc:1514] Processing DeleteTablet for tablet 514eebdf8d6640be835eb36172df00e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:21 UTC) from {username='oryx', principal='oryx/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:46452
I20260620 19:08:21.380044 8847 tablet_replica.cc:330] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1: stopping tablet replica
I20260620 19:08:21.380053 8310 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 19:08:21.380152 8847 raft_consensus.cc:2229] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 19:08:21.380187 8310 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:08:21.380244 8847 raft_consensus.cc:2258] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:21.380522 8845 tablet_replica.cc:330] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696: stopping tablet replica
I20260620 19:08:21.381083 8847 ts_tablet_manager.cc:1905] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:21.381628 8447 tablet_service.cc:1467] Processing CreateTablet for tablet b2bfdc0372da46508d5cfb1ea87803f4 (DEFAULT_TABLE table=test-table [id=7966c39632dd435a945191948d669cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:21.381738 8447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2bfdc0372da46508d5cfb1ea87803f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:21.381845 8727 tablet_service.cc:1467] Processing CreateTablet for tablet b2bfdc0372da46508d5cfb1ea87803f4 (DEFAULT_TABLE table=test-table [id=7966c39632dd435a945191948d669cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:21.381958 8727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2bfdc0372da46508d5cfb1ea87803f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:21.382829 8847 ts_tablet_manager.cc:1918] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:21.382887 8847 log.cc:1196] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/514eebdf8d6640be835eb36172df00e2
I20260620 19:08:21.383172 8821 tablet_bootstrap.cc:492] T b2bfdc0372da46508d5cfb1ea87803f4 P b869b8b864be40fa9446f0e98683c84f: Bootstrap starting.
I20260620 19:08:21.383279 8822 tablet_bootstrap.cc:492] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1: Bootstrap starting.
I20260620 19:08:21.383528 8821 tablet_bootstrap.cc:654] T b2bfdc0372da46508d5cfb1ea87803f4 P b869b8b864be40fa9446f0e98683c84f: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:21.380057 8846 tablet_replica.cc:330] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f: stopping tablet replica
I20260620 19:08:21.381614 8587 tablet_service.cc:1467] Processing CreateTablet for tablet b2bfdc0372da46508d5cfb1ea87803f4 (DEFAULT_TABLE table=test-table [id=7966c39632dd435a945191948d669cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:21.383821 8587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2bfdc0372da46508d5cfb1ea87803f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:21.383936 8822 tablet_bootstrap.cc:654] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:21.384516 8822 tablet_bootstrap.cc:492] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1: No bootstrap required, opened a new log
I20260620 19:08:21.384544 8822 ts_tablet_manager.cc:1397] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:21.384680 8822 raft_consensus.cc:348] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.384742 8822 raft_consensus.cc:374] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:21.384757 8822 raft_consensus.cc:729] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9ddaf367644bacb3aac674f33582c1, State: Initialized, Role: FOLLOWER
I20260620 19:08:21.384811 8822 consensus_queue.cc:260] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.384927 8822 ts_tablet_manager.cc:1428] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:21.385094 8847 ts_tablet_manager.cc:1939] T 514eebdf8d6640be835eb36172df00e2 P 8b9ddaf367644bacb3aac674f33582c1: Deleting consensus metadata
I20260620 19:08:21.385130 8823 tablet_bootstrap.cc:492] T b2bfdc0372da46508d5cfb1ea87803f4 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Bootstrap starting.
I20260620 19:08:21.385501 8823 tablet_bootstrap.cc:654] T b2bfdc0372da46508d5cfb1ea87803f4 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:21.385571 8294 catalog_manager.cc:4841] TS 8b9ddaf367644bacb3aac674f33582c1 (127.0.203.3:35531): tablet 514eebdf8d6640be835eb36172df00e2 (table test-table [id=fb5d899cd43d4581bb4db6b77670d31e]) successfully deleted
I20260620 19:08:21.386094 8823 tablet_bootstrap.cc:492] T b2bfdc0372da46508d5cfb1ea87803f4 P 2073bd7db4d14d9b8bd9ea0c5cb65696: No bootstrap required, opened a new log
I20260620 19:08:21.386132 8823 ts_tablet_manager.cc:1397] T b2bfdc0372da46508d5cfb1ea87803f4 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:08:21.386281 8823 raft_consensus.cc:348] T b2bfdc0372da46508d5cfb1ea87803f4 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.386358 8823 raft_consensus.cc:374] T b2bfdc0372da46508d5cfb1ea87803f4 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:21.386373 8823 raft_consensus.cc:729] T b2bfdc0372da46508d5cfb1ea87803f4 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2073bd7db4d14d9b8bd9ea0c5cb65696, State: Initialized, Role: FOLLOWER
I20260620 19:08:21.386423 8823 consensus_queue.cc:260] T b2bfdc0372da46508d5cfb1ea87803f4 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.386569 8823 ts_tablet_manager.cc:1428] T b2bfdc0372da46508d5cfb1ea87803f4 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:21.384385 8821 tablet_bootstrap.cc:492] T b2bfdc0372da46508d5cfb1ea87803f4 P b869b8b864be40fa9446f0e98683c84f: No bootstrap required, opened a new log
I20260620 19:08:21.386651 8821 ts_tablet_manager.cc:1397] T b2bfdc0372da46508d5cfb1ea87803f4 P b869b8b864be40fa9446f0e98683c84f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 19:08:21.386823 8821 raft_consensus.cc:348] T b2bfdc0372da46508d5cfb1ea87803f4 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.386904 8821 raft_consensus.cc:374] T b2bfdc0372da46508d5cfb1ea87803f4 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:21.386925 8821 raft_consensus.cc:729] T b2bfdc0372da46508d5cfb1ea87803f4 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b869b8b864be40fa9446f0e98683c84f, State: Initialized, Role: FOLLOWER
I20260620 19:08:21.386979 8821 consensus_queue.cc:260] T b2bfdc0372da46508d5cfb1ea87803f4 P b869b8b864be40fa9446f0e98683c84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.387068 8821 ts_tablet_manager.cc:1428] T b2bfdc0372da46508d5cfb1ea87803f4 P b869b8b864be40fa9446f0e98683c84f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:21.387318 8845 raft_consensus.cc:2229] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:21.387434 8845 raft_consensus.cc:2258] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:21.387591 8846 raft_consensus.cc:2229] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:08:21.387739 8846 raft_consensus.cc:2258] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:21.388216 8846 ts_tablet_manager.cc:1905] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:21.388509 8845 ts_tablet_manager.cc:1905] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:21.389577 8846 ts_tablet_manager.cc:1918] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:21.389667 8846 log.cc:1196] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-0/wal/wals/514eebdf8d6640be835eb36172df00e2
I20260620 19:08:21.389899 8846 ts_tablet_manager.cc:1939] T 514eebdf8d6640be835eb36172df00e2 P b869b8b864be40fa9446f0e98683c84f: Deleting consensus metadata
I20260620 19:08:21.390255 8295 catalog_manager.cc:4841] TS b869b8b864be40fa9446f0e98683c84f (127.0.203.1:37653): tablet 514eebdf8d6640be835eb36172df00e2 (table test-table [id=fb5d899cd43d4581bb4db6b77670d31e]) successfully deleted
I20260620 19:08:21.390405 8845 ts_tablet_manager.cc:1918] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:21.390475 8845 log.cc:1196] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/514eebdf8d6640be835eb36172df00e2
I20260620 19:08:21.390798 8845 ts_tablet_manager.cc:1939] T 514eebdf8d6640be835eb36172df00e2 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Deleting consensus metadata
I20260620 19:08:21.391199 8296 catalog_manager.cc:4841] TS 2073bd7db4d14d9b8bd9ea0c5cb65696 (127.0.203.2:40787): tablet 514eebdf8d6640be835eb36172df00e2 (table test-table [id=fb5d899cd43d4581bb4db6b77670d31e]) successfully deleted
I20260620 19:08:21.439178 8827 raft_consensus.cc:482] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:08:21.439299 8827 raft_consensus.cc:504] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.439656 8827 leader_election.cc:288] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b869b8b864be40fa9446f0e98683c84f (127.0.203.1:37653), 2073bd7db4d14d9b8bd9ea0c5cb65696 (127.0.203.2:40787)
I20260620 19:08:21.443192 8820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.439952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:39473 (local address 127.0.203.2:40787)
0620 19:08:21.440001 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:21.440003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:21.440110 (+ 107us) server_negotiation.cc:407] Connection header received
0620 19:08:21.440141 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:21.440142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:21.440151 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:21.440222 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:21.440407 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.441191 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.442281 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.442702 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.442774 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.442785 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:21.443048 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 19:08:21.443096 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 19:08:21.443434 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.439869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:40787 (local address 127.0.203.3:39473)
0620 19:08:21.439997 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:21.440008 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:21.440069 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:21.440247 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:21.440248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:21.440251 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:21.440321 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.440324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.441309 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.441310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.442183 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.442186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.442869 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.442870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.442951 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.442955 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:08:21.443313 (+ 358us) client_negotiation.cc:240] Negotiation successful
0620 19:08:21.443362 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":59,"thread_start_us":65,"threads_started":1}
I20260620 19:08:21.443559 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.440049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:34385 (local address 127.0.203.1:37653)
0620 19:08:21.440092 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:21.440094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:21.440187 (+ 93us) server_negotiation.cc:407] Connection header received
0620 19:08:21.440222 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:21.440223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:21.440232 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:21.440275 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:21.440481 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.441191 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.442605 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.443176 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.443236 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.443246 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:21.443432 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 19:08:21.443467 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:08:21.443773 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.439872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:37653 (local address 127.0.203.3:34385)
0620 19:08:21.440107 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:21.440116 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:21.440174 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:21.440305 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:21.440306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:21.440309 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:21.440367 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.440369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.441288 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.441289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.442509 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.442512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.443280 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.443282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.443356 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.443360 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:08:21.443640 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 19:08:21.443683 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":29,"spinlock_wait_cycles":2816,"thread_start_us":66,"threads_started":1}
I20260620 19:08:21.443763 8606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2bfdc0372da46508d5cfb1ea87803f4" candidate_uuid: "8b9ddaf367644bacb3aac674f33582c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" is_pre_election: true
I20260620 19:08:21.443847 8606 raft_consensus.cc:2454] T b2bfdc0372da46508d5cfb1ea87803f4 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9ddaf367644bacb3aac674f33582c1 in term 0.
I20260620 19:08:21.443998 8465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2bfdc0372da46508d5cfb1ea87803f4" candidate_uuid: "8b9ddaf367644bacb3aac674f33582c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b869b8b864be40fa9446f0e98683c84f" is_pre_election: true
I20260620 19:08:21.444069 8680 leader_election.cc:302] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2073bd7db4d14d9b8bd9ea0c5cb65696, 8b9ddaf367644bacb3aac674f33582c1; no voters:
I20260620 19:08:21.444147 8465 raft_consensus.cc:2454] T b2bfdc0372da46508d5cfb1ea87803f4 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9ddaf367644bacb3aac674f33582c1 in term 0.
I20260620 19:08:21.444185 8827 raft_consensus.cc:2781] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:08:21.444217 8827 raft_consensus.cc:482] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:08:21.444237 8827 raft_consensus.cc:3037] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:21.445087 8827 raft_consensus.cc:504] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.445228 8827 leader_election.cc:288] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [CANDIDATE]: Term 1 election: Requested vote from peers b869b8b864be40fa9446f0e98683c84f (127.0.203.1:37653), 2073bd7db4d14d9b8bd9ea0c5cb65696 (127.0.203.2:40787)
I20260620 19:08:21.445474 8465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2bfdc0372da46508d5cfb1ea87803f4" candidate_uuid: "8b9ddaf367644bacb3aac674f33582c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b869b8b864be40fa9446f0e98683c84f"
I20260620 19:08:21.445559 8465 raft_consensus.cc:3037] T b2bfdc0372da46508d5cfb1ea87803f4 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:21.445547 8606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2bfdc0372da46508d5cfb1ea87803f4" candidate_uuid: "8b9ddaf367644bacb3aac674f33582c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696"
I20260620 19:08:21.445608 8606 raft_consensus.cc:3037] T b2bfdc0372da46508d5cfb1ea87803f4 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:21.446334 8465 raft_consensus.cc:2454] T b2bfdc0372da46508d5cfb1ea87803f4 P b869b8b864be40fa9446f0e98683c84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b9ddaf367644bacb3aac674f33582c1 in term 1.
I20260620 19:08:21.446344 8606 raft_consensus.cc:2454] T b2bfdc0372da46508d5cfb1ea87803f4 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b9ddaf367644bacb3aac674f33582c1 in term 1.
I20260620 19:08:21.446517 8679 leader_election.cc:302] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9ddaf367644bacb3aac674f33582c1, b869b8b864be40fa9446f0e98683c84f; no voters:
I20260620 19:08:21.446594 8827 raft_consensus.cc:2781] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:21.446707 8827 raft_consensus.cc:686] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [term 1 LEADER]: Becoming Leader. State: Replica: 8b9ddaf367644bacb3aac674f33582c1, State: Running, Role: LEADER
I20260620 19:08:21.446776 8827 consensus_queue.cc:237] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.447561 8309 catalog_manager.cc:5495] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b9ddaf367644bacb3aac674f33582c1 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8b9ddaf367644bacb3aac674f33582c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } health_report { overall_health: HEALTHY } } }
Jun 20 19:08:21 dist-test-slave-wd30 krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982500, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.62@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM
I20260620 19:08:21.491654 8861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.483939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38526 (local address 127.0.203.62:36211)
0620 19:08:21.484076 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:21.484078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:21.484151 (+ 73us) server_negotiation.cc:407] Connection header received
0620 19:08:21.484226 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:21.484227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:21.484238 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:21.484325 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:21.484818 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.485478 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.486196 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.486392 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.486475 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.488789 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:21.488790 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:21.488790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:21.488798 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:21.490322 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:21.490771 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:21.490773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:21.490773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:21.490834 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:21.491039 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:21.491040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:21.491040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:21.491213 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:21.491288 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:21.491449 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 19:08:21.491538 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260620 19:08:21.491706 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.483873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36211 (local address 127.0.0.1:38526)
0620 19:08:21.484027 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:21.484073 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:21.484173 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:21.484328 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:21.484329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:21.484687 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:21.484734 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.484736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.485585 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.485586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.486112 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.486113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.486475 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.486476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.486528 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.487083 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:21.487088 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:21.488700 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:21.490457 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:21.490465 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:21.490467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:21.490695 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:21.490915 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:21.490915 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:21.490916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:21.490976 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:21.491277 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:21.491284 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:21.491332 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 19:08:21.491425 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 19:08:21.491591 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":67,"thread_start_us":97,"threads_started":1}
I20260620 19:08:21.493767 8309 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:38526:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 19:08:21.493976 8309 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:08:21.495402 8447 tablet_service.cc:1467] Processing CreateTablet for tablet 38c574d9073e438787ebb9789045b252 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16c1d1d1a3b14f458facc97515f7c796]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:08:21.495412 8727 tablet_service.cc:1467] Processing CreateTablet for tablet 38c574d9073e438787ebb9789045b252 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16c1d1d1a3b14f458facc97515f7c796]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:08:21.495437 8587 tablet_service.cc:1467] Processing CreateTablet for tablet 38c574d9073e438787ebb9789045b252 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16c1d1d1a3b14f458facc97515f7c796]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:08:21.495554 8447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c574d9073e438787ebb9789045b252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:21.495554 8727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c574d9073e438787ebb9789045b252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:21.495575 8587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c574d9073e438787ebb9789045b252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:21.497022 8823 tablet_bootstrap.cc:492] T 38c574d9073e438787ebb9789045b252 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Bootstrap starting.
I20260620 19:08:21.497025 8822 tablet_bootstrap.cc:492] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1: Bootstrap starting.
I20260620 19:08:21.497025 8821 tablet_bootstrap.cc:492] T 38c574d9073e438787ebb9789045b252 P b869b8b864be40fa9446f0e98683c84f: Bootstrap starting.
I20260620 19:08:21.497381 8821 tablet_bootstrap.cc:654] T 38c574d9073e438787ebb9789045b252 P b869b8b864be40fa9446f0e98683c84f: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:21.497409 8822 tablet_bootstrap.cc:654] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:21.497455 8823 tablet_bootstrap.cc:654] T 38c574d9073e438787ebb9789045b252 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:21.498024 8822 tablet_bootstrap.cc:492] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1: No bootstrap required, opened a new log
I20260620 19:08:21.498025 8821 tablet_bootstrap.cc:492] T 38c574d9073e438787ebb9789045b252 P b869b8b864be40fa9446f0e98683c84f: No bootstrap required, opened a new log
I20260620 19:08:21.498073 8821 ts_tablet_manager.cc:1397] T 38c574d9073e438787ebb9789045b252 P b869b8b864be40fa9446f0e98683c84f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:08:21.498073 8822 ts_tablet_manager.cc:1397] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:21.498206 8821 raft_consensus.cc:348] T 38c574d9073e438787ebb9789045b252 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.498206 8822 raft_consensus.cc:348] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.498274 8821 raft_consensus.cc:374] T 38c574d9073e438787ebb9789045b252 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:21.498274 8822 raft_consensus.cc:374] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:21.498296 8822 raft_consensus.cc:729] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9ddaf367644bacb3aac674f33582c1, State: Initialized, Role: FOLLOWER
I20260620 19:08:21.498296 8821 raft_consensus.cc:729] T 38c574d9073e438787ebb9789045b252 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b869b8b864be40fa9446f0e98683c84f, State: Initialized, Role: FOLLOWER
I20260620 19:08:21.498338 8822 consensus_queue.cc:260] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.498338 8821 consensus_queue.cc:260] T 38c574d9073e438787ebb9789045b252 P b869b8b864be40fa9446f0e98683c84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.498421 8822 ts_tablet_manager.cc:1428] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:21.498435 8821 ts_tablet_manager.cc:1428] T 38c574d9073e438787ebb9789045b252 P b869b8b864be40fa9446f0e98683c84f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 19:08:21.498541 8827 tablet_replica.cc:439] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } } }
I20260620 19:08:21.498608 8827 tablet_replica.cc:442] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:08:21.498703 8841 tablet_replica.cc:439] T 38c574d9073e438787ebb9789045b252 P b869b8b864be40fa9446f0e98683c84f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } } }
I20260620 19:08:21.498778 8841 tablet_replica.cc:442] T 38c574d9073e438787ebb9789045b252 P b869b8b864be40fa9446f0e98683c84f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:08:21.499002 8823 tablet_bootstrap.cc:492] T 38c574d9073e438787ebb9789045b252 P 2073bd7db4d14d9b8bd9ea0c5cb65696: No bootstrap required, opened a new log
I20260620 19:08:21.499053 8823 ts_tablet_manager.cc:1397] T 38c574d9073e438787ebb9789045b252 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:08:21.499217 8823 raft_consensus.cc:348] T 38c574d9073e438787ebb9789045b252 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.499284 8823 raft_consensus.cc:374] T 38c574d9073e438787ebb9789045b252 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:21.499301 8823 raft_consensus.cc:729] T 38c574d9073e438787ebb9789045b252 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2073bd7db4d14d9b8bd9ea0c5cb65696, State: Initialized, Role: FOLLOWER
I20260620 19:08:21.499336 8823 consensus_queue.cc:260] T 38c574d9073e438787ebb9789045b252 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.499413 8823 ts_tablet_manager.cc:1428] T 38c574d9073e438787ebb9789045b252 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:21.499481 8829 tablet_replica.cc:439] T 38c574d9073e438787ebb9789045b252 P 2073bd7db4d14d9b8bd9ea0c5cb65696: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } } }
I20260620 19:08:21.499594 8829 tablet_replica.cc:442] T 38c574d9073e438787ebb9789045b252 P 2073bd7db4d14d9b8bd9ea0c5cb65696: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:08:21.513744 8827 raft_consensus.cc:482] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:08:21.513841 8827 raft_consensus.cc:504] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.513994 8827 leader_election.cc:288] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b869b8b864be40fa9446f0e98683c84f (127.0.203.1:37653), 2073bd7db4d14d9b8bd9ea0c5cb65696 (127.0.203.2:40787)
I20260620 19:08:21.514199 8465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c574d9073e438787ebb9789045b252" candidate_uuid: "8b9ddaf367644bacb3aac674f33582c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b869b8b864be40fa9446f0e98683c84f" is_pre_election: true
I20260620 19:08:21.514199 8606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c574d9073e438787ebb9789045b252" candidate_uuid: "8b9ddaf367644bacb3aac674f33582c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" is_pre_election: true
I20260620 19:08:21.514287 8606 raft_consensus.cc:2454] T 38c574d9073e438787ebb9789045b252 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9ddaf367644bacb3aac674f33582c1 in term 0.
I20260620 19:08:21.514287 8465 raft_consensus.cc:2454] T 38c574d9073e438787ebb9789045b252 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9ddaf367644bacb3aac674f33582c1 in term 0.
I20260620 19:08:21.514417 8680 leader_election.cc:302] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2073bd7db4d14d9b8bd9ea0c5cb65696, 8b9ddaf367644bacb3aac674f33582c1; no voters:
I20260620 19:08:21.514541 8827 raft_consensus.cc:2781] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:08:21.514591 8827 raft_consensus.cc:482] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:08:21.514623 8827 raft_consensus.cc:3037] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:21.515297 8827 raft_consensus.cc:504] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.515434 8827 leader_election.cc:288] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1 [CANDIDATE]: Term 1 election: Requested vote from peers b869b8b864be40fa9446f0e98683c84f (127.0.203.1:37653), 2073bd7db4d14d9b8bd9ea0c5cb65696 (127.0.203.2:40787)
I20260620 19:08:21.515620 8465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c574d9073e438787ebb9789045b252" candidate_uuid: "8b9ddaf367644bacb3aac674f33582c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b869b8b864be40fa9446f0e98683c84f"
I20260620 19:08:21.515616 8606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c574d9073e438787ebb9789045b252" candidate_uuid: "8b9ddaf367644bacb3aac674f33582c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696"
I20260620 19:08:21.515689 8465 raft_consensus.cc:3037] T 38c574d9073e438787ebb9789045b252 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:21.515707 8606 raft_consensus.cc:3037] T 38c574d9073e438787ebb9789045b252 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:21.516247 8465 raft_consensus.cc:2454] T 38c574d9073e438787ebb9789045b252 P b869b8b864be40fa9446f0e98683c84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b9ddaf367644bacb3aac674f33582c1 in term 1.
I20260620 19:08:21.516392 8606 raft_consensus.cc:2454] T 38c574d9073e438787ebb9789045b252 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b9ddaf367644bacb3aac674f33582c1 in term 1.
I20260620 19:08:21.516408 8679 leader_election.cc:302] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9ddaf367644bacb3aac674f33582c1, b869b8b864be40fa9446f0e98683c84f; no voters:
I20260620 19:08:21.516502 8827 raft_consensus.cc:2781] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:21.516551 8827 raft_consensus.cc:686] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1 [term 1 LEADER]: Becoming Leader. State: Replica: 8b9ddaf367644bacb3aac674f33582c1, State: Running, Role: LEADER
I20260620 19:08:21.516603 8827 consensus_queue.cc:237] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.516887 8827 tablet_replica.cc:439] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1: TxnStatusTablet state changed. Reason: New leader 8b9ddaf367644bacb3aac674f33582c1. Latest consensus state: current_term: 1 leader_uuid: "8b9ddaf367644bacb3aac674f33582c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } } }
I20260620 19:08:21.516983 8827 tablet_replica.cc:442] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:08:21.517170 8866 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 19:08:21.517198 8310 catalog_manager.cc:5495] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b9ddaf367644bacb3aac674f33582c1 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8b9ddaf367644bacb3aac674f33582c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } health_report { overall_health: HEALTHY } } }
I20260620 19:08:21.517289 8866 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 19:08:21.523471 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.521060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.0.203.3:35531)
0620 19:08:21.521113 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:21.521115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:21.521122 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:08:21.521150 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:21.521155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:21.521169 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:21.521246 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:08:21.521373 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.521844 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.522600 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.522808 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.522889 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.523080 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:08:21.523232 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:08:21.523306 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:21.523362 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 19:08:21.523384 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:08:21.523487 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.520849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:35531 (local address 127.0.0.1:46456)
0620 19:08:21.520935 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:21.520949 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:21.520993 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:21.521252 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:21.521253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:21.521255 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:08:21.521305 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.521307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.521925 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.521927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.522515 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.522516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.522892 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.522893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.522957 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.522978 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:08:21.523306 (+ 328us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:08:21.523315 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 19:08:21.523360 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 19:08:21.523412 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 19:08:21.524595 8606 raft_consensus.cc:1261] T 38c574d9073e438787ebb9789045b252 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 1 FOLLOWER]: Refusing update from remote peer 8b9ddaf367644bacb3aac674f33582c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:21.524681 8465 raft_consensus.cc:1261] T 38c574d9073e438787ebb9789045b252 P b869b8b864be40fa9446f0e98683c84f [term 1 FOLLOWER]: Refusing update from remote peer 8b9ddaf367644bacb3aac674f33582c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:21.524840 8827 consensus_queue.cc:1035] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:21.524947 8853 consensus_queue.cc:1035] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:21.525478 8829 tablet_replica.cc:439] T 38c574d9073e438787ebb9789045b252 P 2073bd7db4d14d9b8bd9ea0c5cb65696: TxnStatusTablet state changed. Reason: New leader 8b9ddaf367644bacb3aac674f33582c1. Latest consensus state: current_term: 1 leader_uuid: "8b9ddaf367644bacb3aac674f33582c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } } }
I20260620 19:08:21.525568 8829 tablet_replica.cc:442] T 38c574d9073e438787ebb9789045b252 P 2073bd7db4d14d9b8bd9ea0c5cb65696: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:08:21.525701 8841 tablet_replica.cc:439] T 38c574d9073e438787ebb9789045b252 P b869b8b864be40fa9446f0e98683c84f: TxnStatusTablet state changed. Reason: New leader 8b9ddaf367644bacb3aac674f33582c1. Latest consensus state: current_term: 1 leader_uuid: "8b9ddaf367644bacb3aac674f33582c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } } }
I20260620 19:08:21.525786 8841 tablet_replica.cc:442] T 38c574d9073e438787ebb9789045b252 P b869b8b864be40fa9446f0e98683c84f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:08:21.526336 8827 tablet_replica.cc:439] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b9ddaf367644bacb3aac674f33582c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } } }
I20260620 19:08:21.526414 8827 tablet_replica.cc:442] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:08:21.526968 8829 tablet_replica.cc:439] T 38c574d9073e438787ebb9789045b252 P 2073bd7db4d14d9b8bd9ea0c5cb65696: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b9ddaf367644bacb3aac674f33582c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } } }
I20260620 19:08:21.527053 8829 tablet_replica.cc:442] T 38c574d9073e438787ebb9789045b252 P 2073bd7db4d14d9b8bd9ea0c5cb65696: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:08:21.527838 8853 tablet_replica.cc:439] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b9ddaf367644bacb3aac674f33582c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } } }
I20260620 19:08:21.527957 8853 tablet_replica.cc:442] T 38c574d9073e438787ebb9789045b252 P 8b9ddaf367644bacb3aac674f33582c1: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:08:21.528465 8841 tablet_replica.cc:439] T 38c574d9073e438787ebb9789045b252 P b869b8b864be40fa9446f0e98683c84f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b9ddaf367644bacb3aac674f33582c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } } }
I20260620 19:08:21.528553 8841 tablet_replica.cc:442] T 38c574d9073e438787ebb9789045b252 P b869b8b864be40fa9446f0e98683c84f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:08:21.528694 8835 mvcc.cc:205] Tried to move back new op lower bound from 7299000326243020800 to 7299000326212349952. Current Snapshot: MvccSnapshot[applied={T|T < 7299000326243020800 or (T in {7299000326243020800})}]
I20260620 19:08:21.530515 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.528096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.0.203.3:35531)
0620 19:08:21.528129 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:21.528131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:21.528138 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:08:21.528156 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:21.528157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:21.528179 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:21.528243 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:08:21.528417 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.529119 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.529813 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.530144 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.530215 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.530312 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:08:21.530350 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:08:21.530400 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:21.530435 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 19:08:21.530451 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:08:21.535451 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.533036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:51037 (local address 127.0.203.3:35531)
0620 19:08:21.533071 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:21.533073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:21.533082 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:08:21.533103 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:21.533104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:21.533112 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:21.533165 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:08:21.533281 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.533972 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.534697 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.534987 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.535055 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.535230 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:08:21.535259 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:08:21.535307 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:21.535357 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 19:08:21.535372 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:08:21.535461 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.532866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:35531 (local address 127.0.203.3:51037)
0620 19:08:21.532980 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:21.533011 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:21.533064 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:21.533166 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:21.533167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:21.533168 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:08:21.533223 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.533225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.534045 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.534045 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.534621 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.534623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.535059 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.535060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.535103 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.535115 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:08:21.535314 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:08:21.535316 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:08:21.535349 (+ 33us) client_negotiation.cc:240] Negotiation successful
0620 19:08:21.535382 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":67,"threads_started":1}
I20260620 19:08:21.537328 8465 raft_consensus.cc:1261] T b2bfdc0372da46508d5cfb1ea87803f4 P b869b8b864be40fa9446f0e98683c84f [term 1 FOLLOWER]: Refusing update from remote peer 8b9ddaf367644bacb3aac674f33582c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:21.537376 8606 raft_consensus.cc:1261] T b2bfdc0372da46508d5cfb1ea87803f4 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 1 FOLLOWER]: Refusing update from remote peer 8b9ddaf367644bacb3aac674f33582c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:21.537556 8871 consensus_queue.cc:1035] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:21.537659 8853 consensus_queue.cc:1035] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:21.549124 8310 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:21.549223 8310 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:21.549829 8310 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354: Sending DeleteTablet for 3 replicas of tablet b2bfdc0372da46508d5cfb1ea87803f4
I20260620 19:08:21.550055 8447 tablet_service.cc:1514] Processing DeleteTablet for tablet b2bfdc0372da46508d5cfb1ea87803f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:21 UTC) from {username='oryx', principal='oryx/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:37306
I20260620 19:08:21.550177 8846 tablet_replica.cc:330] T b2bfdc0372da46508d5cfb1ea87803f4 P b869b8b864be40fa9446f0e98683c84f: stopping tablet replica
I20260620 19:08:21.550205 8727 tablet_service.cc:1514] Processing DeleteTablet for tablet b2bfdc0372da46508d5cfb1ea87803f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:21 UTC) from {username='oryx', principal='oryx/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:46452
I20260620 19:08:21.550238 8846 raft_consensus.cc:2229] T b2bfdc0372da46508d5cfb1ea87803f4 P b869b8b864be40fa9446f0e98683c84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:21.550282 8846 raft_consensus.cc:2258] T b2bfdc0372da46508d5cfb1ea87803f4 P b869b8b864be40fa9446f0e98683c84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:21.550289 8847 tablet_replica.cc:330] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1: stopping tablet replica
I20260620 19:08:21.550343 8847 raft_consensus.cc:2229] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:08:21.550413 8847 raft_consensus.cc:2258] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:21.551409 8587 tablet_service.cc:1514] Processing DeleteTablet for tablet b2bfdc0372da46508d5cfb1ea87803f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:21 UTC) from {username='oryx', principal='oryx/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:60452
I20260620 19:08:21.551502 8845 tablet_replica.cc:330] T b2bfdc0372da46508d5cfb1ea87803f4 P 2073bd7db4d14d9b8bd9ea0c5cb65696: stopping tablet replica
I20260620 19:08:21.551522 8846 ts_tablet_manager.cc:1905] T b2bfdc0372da46508d5cfb1ea87803f4 P b869b8b864be40fa9446f0e98683c84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:21.551540 8845 raft_consensus.cc:2229] T b2bfdc0372da46508d5cfb1ea87803f4 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:21.551571 8845 raft_consensus.cc:2258] T b2bfdc0372da46508d5cfb1ea87803f4 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:21.551854 8845 ts_tablet_manager.cc:1905] T b2bfdc0372da46508d5cfb1ea87803f4 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:21.552623 8847 ts_tablet_manager.cc:1905] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:21.553231 8846 ts_tablet_manager.cc:1918] T b2bfdc0372da46508d5cfb1ea87803f4 P b869b8b864be40fa9446f0e98683c84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:08:21.553278 8846 log.cc:1196] T b2bfdc0372da46508d5cfb1ea87803f4 P b869b8b864be40fa9446f0e98683c84f: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-0/wal/wals/b2bfdc0372da46508d5cfb1ea87803f4
I20260620 19:08:21.553298 8845 ts_tablet_manager.cc:1918] T b2bfdc0372da46508d5cfb1ea87803f4 P 2073bd7db4d14d9b8bd9ea0c5cb65696: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:08:21.553364 8845 log.cc:1196] T b2bfdc0372da46508d5cfb1ea87803f4 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/b2bfdc0372da46508d5cfb1ea87803f4
I20260620 19:08:21.553504 8846 ts_tablet_manager.cc:1939] T b2bfdc0372da46508d5cfb1ea87803f4 P b869b8b864be40fa9446f0e98683c84f: Deleting consensus metadata
I20260620 19:08:21.553578 8845 ts_tablet_manager.cc:1939] T b2bfdc0372da46508d5cfb1ea87803f4 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Deleting consensus metadata
I20260620 19:08:21.553853 8296 catalog_manager.cc:4841] TS 2073bd7db4d14d9b8bd9ea0c5cb65696 (127.0.203.2:40787): tablet b2bfdc0372da46508d5cfb1ea87803f4 (table test-table [id=7966c39632dd435a945191948d669cb7]) successfully deleted
I20260620 19:08:21.554208 8847 ts_tablet_manager.cc:1918] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:08:21.554244 8847 log.cc:1196] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/b2bfdc0372da46508d5cfb1ea87803f4
I20260620 19:08:21.554389 8847 ts_tablet_manager.cc:1939] T b2bfdc0372da46508d5cfb1ea87803f4 P 8b9ddaf367644bacb3aac674f33582c1: Deleting consensus metadata
I20260620 19:08:21.554561 8295 catalog_manager.cc:4841] TS b869b8b864be40fa9446f0e98683c84f (127.0.203.1:37653): tablet b2bfdc0372da46508d5cfb1ea87803f4 (table test-table [id=7966c39632dd435a945191948d669cb7]) successfully deleted
I20260620 19:08:21.555740 8294 catalog_manager.cc:4841] TS 8b9ddaf367644bacb3aac674f33582c1 (127.0.203.3:35531): tablet b2bfdc0372da46508d5cfb1ea87803f4 (table test-table [id=7966c39632dd435a945191948d669cb7]) successfully deleted
Jun 20 19:08:21 dist-test-slave-wd30 krb5kdc[8265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:08:21 dist-test-slave-wd30 krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:21 dist-test-slave-wd30 krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:08:21.580077 8861 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:21.573738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.0.203.62:36211)
0620 19:08:21.573800 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:21.573802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:21.573898 (+ 96us) server_negotiation.cc:407] Connection header received
0620 19:08:21.573967 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:21.573968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:21.573980 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:21.574028 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:21.574727 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.575346 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.576487 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.576751 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.576832 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.580009 (+ 3177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38536: BlockingRecv error: recv got EOF from 127.0.0.1:38536 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 20 19:08:21 dist-test-slave-wd30 krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM
I20260620 19:08:21.588846 8861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.580923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38548 (local address 127.0.203.62:36211)
0620 19:08:21.580978 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:21.580981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:21.581030 (+ 49us) server_negotiation.cc:407] Connection header received
0620 19:08:21.581096 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:21.581097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:21.581107 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:21.581170 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:21.581736 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.582488 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.583474 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.583784 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.583854 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.586165 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:21.586166 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:21.586166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:21.586174 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:21.587762 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:21.588083 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:21.588085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:21.588085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:21.588143 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:21.588314 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:21.588315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:21.588315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:21.588448 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:21.588569 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:21.588675 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 19:08:21.588753 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:08:21.590379 8310 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:08:21.590534 8310 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:08:21.591969 8447 tablet_service.cc:1467] Processing CreateTablet for tablet 17a02cdc91cc4c1586c6845eac98096a (DEFAULT_TABLE table=test-table [id=b228213b41eb417b86c36ae970c1a704]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:21.591990 8587 tablet_service.cc:1467] Processing CreateTablet for tablet 17a02cdc91cc4c1586c6845eac98096a (DEFAULT_TABLE table=test-table [id=b228213b41eb417b86c36ae970c1a704]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:21.592067 8727 tablet_service.cc:1467] Processing CreateTablet for tablet 17a02cdc91cc4c1586c6845eac98096a (DEFAULT_TABLE table=test-table [id=b228213b41eb417b86c36ae970c1a704]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:21.592113 8587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a02cdc91cc4c1586c6845eac98096a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:21.592113 8447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a02cdc91cc4c1586c6845eac98096a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:21.592170 8727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a02cdc91cc4c1586c6845eac98096a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:21.593420 8823 tablet_bootstrap.cc:492] T 17a02cdc91cc4c1586c6845eac98096a P 2073bd7db4d14d9b8bd9ea0c5cb65696: Bootstrap starting.
I20260620 19:08:21.593461 8822 tablet_bootstrap.cc:492] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1: Bootstrap starting.
I20260620 19:08:21.593763 8822 tablet_bootstrap.cc:654] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:21.593802 8823 tablet_bootstrap.cc:654] T 17a02cdc91cc4c1586c6845eac98096a P 2073bd7db4d14d9b8bd9ea0c5cb65696: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:21.594127 8821 tablet_bootstrap.cc:492] T 17a02cdc91cc4c1586c6845eac98096a P b869b8b864be40fa9446f0e98683c84f: Bootstrap starting.
I20260620 19:08:21.594405 8821 tablet_bootstrap.cc:654] T 17a02cdc91cc4c1586c6845eac98096a P b869b8b864be40fa9446f0e98683c84f: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:21.594434 8822 tablet_bootstrap.cc:492] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1: No bootstrap required, opened a new log
I20260620 19:08:21.594470 8822 ts_tablet_manager.cc:1397] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:21.594606 8822 raft_consensus.cc:348] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.594684 8822 raft_consensus.cc:374] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:21.594707 8822 raft_consensus.cc:729] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9ddaf367644bacb3aac674f33582c1, State: Initialized, Role: FOLLOWER
I20260620 19:08:21.594755 8822 consensus_queue.cc:260] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.594885 8822 ts_tablet_manager.cc:1428] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:21.595274 8821 tablet_bootstrap.cc:492] T 17a02cdc91cc4c1586c6845eac98096a P b869b8b864be40fa9446f0e98683c84f: No bootstrap required, opened a new log
I20260620 19:08:21.595324 8821 ts_tablet_manager.cc:1397] T 17a02cdc91cc4c1586c6845eac98096a P b869b8b864be40fa9446f0e98683c84f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:08:21.595431 8821 raft_consensus.cc:348] T 17a02cdc91cc4c1586c6845eac98096a P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.595486 8821 raft_consensus.cc:374] T 17a02cdc91cc4c1586c6845eac98096a P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:21.595503 8821 raft_consensus.cc:729] T 17a02cdc91cc4c1586c6845eac98096a P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b869b8b864be40fa9446f0e98683c84f, State: Initialized, Role: FOLLOWER
I20260620 19:08:21.595541 8821 consensus_queue.cc:260] T 17a02cdc91cc4c1586c6845eac98096a P b869b8b864be40fa9446f0e98683c84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.595620 8821 ts_tablet_manager.cc:1428] T 17a02cdc91cc4c1586c6845eac98096a P b869b8b864be40fa9446f0e98683c84f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:21.595947 8823 tablet_bootstrap.cc:492] T 17a02cdc91cc4c1586c6845eac98096a P 2073bd7db4d14d9b8bd9ea0c5cb65696: No bootstrap required, opened a new log
I20260620 19:08:21.596019 8823 ts_tablet_manager.cc:1397] T 17a02cdc91cc4c1586c6845eac98096a P 2073bd7db4d14d9b8bd9ea0c5cb65696: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 19:08:21.596168 8823 raft_consensus.cc:348] T 17a02cdc91cc4c1586c6845eac98096a P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.596239 8823 raft_consensus.cc:374] T 17a02cdc91cc4c1586c6845eac98096a P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:21.596258 8823 raft_consensus.cc:729] T 17a02cdc91cc4c1586c6845eac98096a P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2073bd7db4d14d9b8bd9ea0c5cb65696, State: Initialized, Role: FOLLOWER
I20260620 19:08:21.596294 8823 consensus_queue.cc:260] T 17a02cdc91cc4c1586c6845eac98096a P 2073bd7db4d14d9b8bd9ea0c5cb65696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.596395 8823 ts_tablet_manager.cc:1428] T 17a02cdc91cc4c1586c6845eac98096a P 2073bd7db4d14d9b8bd9ea0c5cb65696: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 19:08:21.606033 8871 raft_consensus.cc:482] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:08:21.606142 8871 raft_consensus.cc:504] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.606323 8871 leader_election.cc:288] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b869b8b864be40fa9446f0e98683c84f (127.0.203.1:37653), 2073bd7db4d14d9b8bd9ea0c5cb65696 (127.0.203.2:40787)
I20260620 19:08:21.606475 8465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a02cdc91cc4c1586c6845eac98096a" candidate_uuid: "8b9ddaf367644bacb3aac674f33582c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b869b8b864be40fa9446f0e98683c84f" is_pre_election: true
I20260620 19:08:21.606598 8465 raft_consensus.cc:2454] T 17a02cdc91cc4c1586c6845eac98096a P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9ddaf367644bacb3aac674f33582c1 in term 0.
I20260620 19:08:21.606644 8606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a02cdc91cc4c1586c6845eac98096a" candidate_uuid: "8b9ddaf367644bacb3aac674f33582c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" is_pre_election: true
I20260620 19:08:21.606730 8679 leader_election.cc:302] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9ddaf367644bacb3aac674f33582c1, b869b8b864be40fa9446f0e98683c84f; no voters:
I20260620 19:08:21.606745 8606 raft_consensus.cc:2454] T 17a02cdc91cc4c1586c6845eac98096a P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9ddaf367644bacb3aac674f33582c1 in term 0.
I20260620 19:08:21.606827 8871 raft_consensus.cc:2781] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:08:21.606860 8871 raft_consensus.cc:482] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:08:21.606884 8871 raft_consensus.cc:3037] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:21.607669 8871 raft_consensus.cc:504] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.607805 8871 leader_election.cc:288] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [CANDIDATE]: Term 1 election: Requested vote from peers b869b8b864be40fa9446f0e98683c84f (127.0.203.1:37653), 2073bd7db4d14d9b8bd9ea0c5cb65696 (127.0.203.2:40787)
I20260620 19:08:21.607971 8606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a02cdc91cc4c1586c6845eac98096a" candidate_uuid: "8b9ddaf367644bacb3aac674f33582c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696"
I20260620 19:08:21.608052 8606 raft_consensus.cc:3037] T 17a02cdc91cc4c1586c6845eac98096a P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:21.608840 8606 raft_consensus.cc:2454] T 17a02cdc91cc4c1586c6845eac98096a P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b9ddaf367644bacb3aac674f33582c1 in term 1.
I20260620 19:08:21.608983 8465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a02cdc91cc4c1586c6845eac98096a" candidate_uuid: "8b9ddaf367644bacb3aac674f33582c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b869b8b864be40fa9446f0e98683c84f"
I20260620 19:08:21.609064 8465 raft_consensus.cc:3037] T 17a02cdc91cc4c1586c6845eac98096a P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:21.609912 8465 raft_consensus.cc:2454] T 17a02cdc91cc4c1586c6845eac98096a P b869b8b864be40fa9446f0e98683c84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b9ddaf367644bacb3aac674f33582c1 in term 1.
I20260620 19:08:21.610090 8680 leader_election.cc:302] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2073bd7db4d14d9b8bd9ea0c5cb65696, 8b9ddaf367644bacb3aac674f33582c1; no voters:
I20260620 19:08:21.610188 8871 raft_consensus.cc:2781] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:21.610239 8871 raft_consensus.cc:686] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [term 1 LEADER]: Becoming Leader. State: Replica: 8b9ddaf367644bacb3aac674f33582c1, State: Running, Role: LEADER
I20260620 19:08:21.610296 8871 consensus_queue.cc:237] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.610831 8310 catalog_manager.cc:5495] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b9ddaf367644bacb3aac674f33582c1 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8b9ddaf367644bacb3aac674f33582c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } health_report { overall_health: HEALTHY } } }
I20260620 19:08:21.619452 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.617119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46474 (local address 127.0.203.3:35531)
0620 19:08:21.617186 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:21.617188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:21.617196 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:08:21.617219 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:21.617220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:21.617229 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:21.617284 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:08:21.617440 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.617915 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.618686 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.618898 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.618989 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.619184 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:08:21.619219 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:08:21.619290 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:21.619351 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 19:08:21.619370 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:08:21.620354 8465 raft_consensus.cc:1261] T 17a02cdc91cc4c1586c6845eac98096a P b869b8b864be40fa9446f0e98683c84f [term 1 FOLLOWER]: Refusing update from remote peer 8b9ddaf367644bacb3aac674f33582c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:21.620396 8606 raft_consensus.cc:1261] T 17a02cdc91cc4c1586c6845eac98096a P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 1 FOLLOWER]: Refusing update from remote peer 8b9ddaf367644bacb3aac674f33582c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:21.620599 8871 consensus_queue.cc:1035] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:21.620689 8875 consensus_queue.cc:1035] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:21.623245 8310 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:21.623337 8310 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:21.624037 8310 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354: Sending DeleteTablet for 3 replicas of tablet 17a02cdc91cc4c1586c6845eac98096a
I20260620 19:08:21.624289 8587 tablet_service.cc:1514] Processing DeleteTablet for tablet 17a02cdc91cc4c1586c6845eac98096a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:21 UTC) from {username='oryx', principal='oryx/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:60452
I20260620 19:08:21.624325 8727 tablet_service.cc:1514] Processing DeleteTablet for tablet 17a02cdc91cc4c1586c6845eac98096a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:21 UTC) from {username='oryx', principal='oryx/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:46452
I20260620 19:08:21.624374 8845 tablet_replica.cc:330] T 17a02cdc91cc4c1586c6845eac98096a P 2073bd7db4d14d9b8bd9ea0c5cb65696: stopping tablet replica
I20260620 19:08:21.624400 8847 tablet_replica.cc:330] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1: stopping tablet replica
I20260620 19:08:21.624418 8845 raft_consensus.cc:2229] T 17a02cdc91cc4c1586c6845eac98096a P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:21.624452 8847 raft_consensus.cc:2229] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:08:21.624459 8845 raft_consensus.cc:2258] T 17a02cdc91cc4c1586c6845eac98096a P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:21.624512 8847 raft_consensus.cc:2258] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:21.624708 8447 tablet_service.cc:1514] Processing DeleteTablet for tablet 17a02cdc91cc4c1586c6845eac98096a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:21 UTC) from {username='oryx', principal='oryx/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:37306
I20260620 19:08:21.624714 8446 tablet_service.cc:1514] Processing DeleteTablet for tablet 17a02cdc91cc4c1586c6845eac98096a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:21 UTC) from {username='oryx', principal='oryx/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:37306
I20260620 19:08:21.624802 8846 tablet_replica.cc:330] T 17a02cdc91cc4c1586c6845eac98096a P b869b8b864be40fa9446f0e98683c84f: stopping tablet replica
I20260620 19:08:21.624852 8846 raft_consensus.cc:2229] T 17a02cdc91cc4c1586c6845eac98096a P b869b8b864be40fa9446f0e98683c84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:21.624898 8846 raft_consensus.cc:2258] T 17a02cdc91cc4c1586c6845eac98096a P b869b8b864be40fa9446f0e98683c84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:21.625067 8845 ts_tablet_manager.cc:1905] T 17a02cdc91cc4c1586c6845eac98096a P 2073bd7db4d14d9b8bd9ea0c5cb65696: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:21.625161 8847 ts_tablet_manager.cc:1905] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:21.625187 8310 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:08:21.625312 8310 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:08:21.626406 8727 tablet_service.cc:1467] Processing CreateTablet for tablet 1a20efa5ee4f48158e5bd2a593f6b0d1 (DEFAULT_TABLE table=test-table [id=e24992f80b3a4f08908bf3a18dadd94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:21.626544 8727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a20efa5ee4f48158e5bd2a593f6b0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:21.626614 8845 ts_tablet_manager.cc:1918] T 17a02cdc91cc4c1586c6845eac98096a P 2073bd7db4d14d9b8bd9ea0c5cb65696: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:21.626655 8845 log.cc:1196] T 17a02cdc91cc4c1586c6845eac98096a P 2073bd7db4d14d9b8bd9ea0c5cb65696: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/17a02cdc91cc4c1586c6845eac98096a
I20260620 19:08:21.626685 8847 ts_tablet_manager.cc:1918] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:21.626739 8847 log.cc:1196] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/17a02cdc91cc4c1586c6845eac98096a
I20260620 19:08:21.626873 8845 ts_tablet_manager.cc:1939] T 17a02cdc91cc4c1586c6845eac98096a P 2073bd7db4d14d9b8bd9ea0c5cb65696: Deleting consensus metadata
I20260620 19:08:21.626964 8847 ts_tablet_manager.cc:1939] T 17a02cdc91cc4c1586c6845eac98096a P 8b9ddaf367644bacb3aac674f33582c1: Deleting consensus metadata
I20260620 19:08:21.627285 8294 catalog_manager.cc:4841] TS 8b9ddaf367644bacb3aac674f33582c1 (127.0.203.3:35531): tablet 17a02cdc91cc4c1586c6845eac98096a (table test-table [id=b228213b41eb417b86c36ae970c1a704]) successfully deleted
I20260620 19:08:21.627702 8846 ts_tablet_manager.cc:1905] T 17a02cdc91cc4c1586c6845eac98096a P b869b8b864be40fa9446f0e98683c84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:21.627919 8822 tablet_bootstrap.cc:492] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 8b9ddaf367644bacb3aac674f33582c1: Bootstrap starting.
I20260620 19:08:21.628028 8446 tablet_service.cc:1467] Processing CreateTablet for tablet 1a20efa5ee4f48158e5bd2a593f6b0d1 (DEFAULT_TABLE table=test-table [id=e24992f80b3a4f08908bf3a18dadd94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:21.628122 8446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a20efa5ee4f48158e5bd2a593f6b0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:21.628314 8822 tablet_bootstrap.cc:654] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 8b9ddaf367644bacb3aac674f33582c1: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:21.629088 8846 ts_tablet_manager.cc:1918] T 17a02cdc91cc4c1586c6845eac98096a P b869b8b864be40fa9446f0e98683c84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:21.629135 8846 log.cc:1196] T 17a02cdc91cc4c1586c6845eac98096a P b869b8b864be40fa9446f0e98683c84f: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-0/wal/wals/17a02cdc91cc4c1586c6845eac98096a
I20260620 19:08:21.629335 8846 ts_tablet_manager.cc:1939] T 17a02cdc91cc4c1586c6845eac98096a P b869b8b864be40fa9446f0e98683c84f: Deleting consensus metadata
I20260620 19:08:21.629510 8821 tablet_bootstrap.cc:492] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P b869b8b864be40fa9446f0e98683c84f: Bootstrap starting.
I20260620 19:08:21.629716 8296 catalog_manager.cc:4841] TS 2073bd7db4d14d9b8bd9ea0c5cb65696 (127.0.203.2:40787): tablet 17a02cdc91cc4c1586c6845eac98096a (table test-table [id=b228213b41eb417b86c36ae970c1a704]) successfully deleted
I20260620 19:08:21.629801 8821 tablet_bootstrap.cc:654] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P b869b8b864be40fa9446f0e98683c84f: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:21.629869 8587 tablet_service.cc:1467] Processing CreateTablet for tablet 1a20efa5ee4f48158e5bd2a593f6b0d1 (DEFAULT_TABLE table=test-table [id=e24992f80b3a4f08908bf3a18dadd94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:21.629969 8587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a20efa5ee4f48158e5bd2a593f6b0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:21.630323 8295 catalog_manager.cc:4841] TS b869b8b864be40fa9446f0e98683c84f (127.0.203.1:37653): tablet 17a02cdc91cc4c1586c6845eac98096a (table test-table [id=b228213b41eb417b86c36ae970c1a704]) successfully deleted
W20260620 19:08:21.630448 8295 catalog_manager.cc:4808] TS b869b8b864be40fa9446f0e98683c84f (127.0.203.1:37653): delete failed for tablet 17a02cdc91cc4c1586c6845eac98096a because the tablet was not found. No further retry: Not found: Tablet not found: 17a02cdc91cc4c1586c6845eac98096a
I20260620 19:08:21.630617 8822 tablet_bootstrap.cc:492] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 8b9ddaf367644bacb3aac674f33582c1: No bootstrap required, opened a new log
I20260620 19:08:21.630662 8822 ts_tablet_manager.cc:1397] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 8b9ddaf367644bacb3aac674f33582c1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 19:08:21.630826 8822 raft_consensus.cc:348] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.630889 8822 raft_consensus.cc:374] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:21.630908 8822 raft_consensus.cc:729] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9ddaf367644bacb3aac674f33582c1, State: Initialized, Role: FOLLOWER
I20260620 19:08:21.630952 8822 consensus_queue.cc:260] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 8b9ddaf367644bacb3aac674f33582c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.631049 8822 ts_tablet_manager.cc:1428] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 8b9ddaf367644bacb3aac674f33582c1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:21.631227 8823 tablet_bootstrap.cc:492] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Bootstrap starting.
I20260620 19:08:21.631649 8823 tablet_bootstrap.cc:654] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:21.632449 8821 tablet_bootstrap.cc:492] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P b869b8b864be40fa9446f0e98683c84f: No bootstrap required, opened a new log
I20260620 19:08:21.632509 8821 ts_tablet_manager.cc:1397] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P b869b8b864be40fa9446f0e98683c84f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 19:08:21.632586 8823 tablet_bootstrap.cc:492] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696: No bootstrap required, opened a new log
I20260620 19:08:21.632635 8823 ts_tablet_manager.cc:1397] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:08:21.632614 8821 raft_consensus.cc:348] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.632681 8821 raft_consensus.cc:374] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:21.632707 8821 raft_consensus.cc:729] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b869b8b864be40fa9446f0e98683c84f, State: Initialized, Role: FOLLOWER
I20260620 19:08:21.632748 8821 consensus_queue.cc:260] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P b869b8b864be40fa9446f0e98683c84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.632784 8823 raft_consensus.cc:348] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.632849 8821 ts_tablet_manager.cc:1428] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P b869b8b864be40fa9446f0e98683c84f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 19:08:21.632858 8823 raft_consensus.cc:374] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:21.632882 8823 raft_consensus.cc:729] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2073bd7db4d14d9b8bd9ea0c5cb65696, State: Initialized, Role: FOLLOWER
I20260620 19:08:21.632948 8823 consensus_queue.cc:260] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.633038 8823 ts_tablet_manager.cc:1428] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:21.713678 8829 raft_consensus.cc:482] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:08:21.713799 8829 raft_consensus.cc:504] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.714174 8829 leader_election.cc:288] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b869b8b864be40fa9446f0e98683c84f (127.0.203.1:37653), 8b9ddaf367644bacb3aac674f33582c1 (127.0.203.3:35531)
I20260620 19:08:21.717801 8902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.714384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:35531 (local address 127.0.203.2:58195)
0620 19:08:21.714522 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:21.714534 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:21.714597 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:21.714742 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:21.714742 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:21.714745 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:21.714833 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.714835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.715899 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.715900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.716772 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.716774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.717517 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.717519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.717591 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.717596 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:08:21.717656 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 19:08:21.717694 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":46,"thread_start_us":66,"threads_started":1}
I20260620 19:08:21.717801 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.714577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:58195 (local address 127.0.203.3:35531)
0620 19:08:21.714616 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:21.714618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:21.714625 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:08:21.714656 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:21.714657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:21.714666 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:21.714736 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:21.714955 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.715760 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.716932 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.717405 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.717478 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.717489 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:21.717676 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 19:08:21.717713 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:08:21.718146 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.714384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:37653 (local address 127.0.203.2:49125)
0620 19:08:21.714569 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:21.714578 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:21.714655 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:21.714823 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:21.714824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:21.714826 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:21.714898 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.714900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.715842 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.715843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.717083 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.717086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.717833 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.717834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.717909 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.717912 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:08:21.717994 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 19:08:21.718043 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":30,"thread_start_us":77,"threads_started":1}
I20260620 19:08:21.718253 8747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a20efa5ee4f48158e5bd2a593f6b0d1" candidate_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9ddaf367644bacb3aac674f33582c1" is_pre_election: true
I20260620 19:08:21.718340 8747 raft_consensus.cc:2454] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2073bd7db4d14d9b8bd9ea0c5cb65696 in term 0.
I20260620 19:08:21.718353 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.714490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:49125 (local address 127.0.203.1:37653)
0620 19:08:21.714537 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:21.714539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:21.714629 (+ 90us) server_negotiation.cc:407] Connection header received
0620 19:08:21.714737 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:21.714739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:21.714754 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:21.714819 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:21.715014 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.715744 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.717187 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.717713 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.717789 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.717799 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:21.718226 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 19:08:21.718276 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:08:21.718524 8465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a20efa5ee4f48158e5bd2a593f6b0d1" candidate_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b869b8b864be40fa9446f0e98683c84f" is_pre_election: true
I20260620 19:08:21.718600 8465 raft_consensus.cc:2454] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2073bd7db4d14d9b8bd9ea0c5cb65696 in term 0.
I20260620 19:08:21.718564 8538 leader_election.cc:302] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2073bd7db4d14d9b8bd9ea0c5cb65696, 8b9ddaf367644bacb3aac674f33582c1; no voters:
I20260620 19:08:21.718683 8829 raft_consensus.cc:2781] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:08:21.718712 8829 raft_consensus.cc:482] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:08:21.718740 8829 raft_consensus.cc:3037] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:21.719460 8829 raft_consensus.cc:504] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.719615 8829 leader_election.cc:288] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [CANDIDATE]: Term 1 election: Requested vote from peers b869b8b864be40fa9446f0e98683c84f (127.0.203.1:37653), 8b9ddaf367644bacb3aac674f33582c1 (127.0.203.3:35531)
I20260620 19:08:21.719751 8465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a20efa5ee4f48158e5bd2a593f6b0d1" candidate_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b869b8b864be40fa9446f0e98683c84f"
I20260620 19:08:21.719825 8465 raft_consensus.cc:3037] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P b869b8b864be40fa9446f0e98683c84f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:21.719799 8747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a20efa5ee4f48158e5bd2a593f6b0d1" candidate_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9ddaf367644bacb3aac674f33582c1"
I20260620 19:08:21.719856 8747 raft_consensus.cc:3037] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 8b9ddaf367644bacb3aac674f33582c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:21.720592 8747 raft_consensus.cc:2454] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 8b9ddaf367644bacb3aac674f33582c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2073bd7db4d14d9b8bd9ea0c5cb65696 in term 1.
I20260620 19:08:21.720652 8465 raft_consensus.cc:2454] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P b869b8b864be40fa9446f0e98683c84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2073bd7db4d14d9b8bd9ea0c5cb65696 in term 1.
I20260620 19:08:21.720767 8538 leader_election.cc:302] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2073bd7db4d14d9b8bd9ea0c5cb65696, 8b9ddaf367644bacb3aac674f33582c1; no voters:
I20260620 19:08:21.720863 8829 raft_consensus.cc:2781] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:21.721016 8829 raft_consensus.cc:686] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 1 LEADER]: Becoming Leader. State: Replica: 2073bd7db4d14d9b8bd9ea0c5cb65696, State: Running, Role: LEADER
I20260620 19:08:21.721086 8829 consensus_queue.cc:237] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } }
I20260620 19:08:21.721850 8310 catalog_manager.cc:5495] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2073bd7db4d14d9b8bd9ea0c5cb65696 (127.0.203.2). New cstate: current_term: 1 leader_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2073bd7db4d14d9b8bd9ea0c5cb65696" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 40787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 } health_report { overall_health: UNKNOWN } } }
I20260620 19:08:21.729002 8820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.726575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60458 (local address 127.0.203.2:40787)
0620 19:08:21.726618 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:21.726620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:21.726629 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:08:21.726704 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:21.726704 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:21.726722 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:21.726812 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:08:21.726932 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.727464 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.728226 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.728423 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.728505 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.728671 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:08:21.728773 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:08:21.728849 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:21.728910 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 19:08:21.728927 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:08:21.735985 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.733676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:53775 (local address 127.0.203.3:35531)
0620 19:08:21.733721 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:21.733723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:21.733758 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:08:21.733832 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:21.733833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:21.733842 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:21.733894 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:08:21.733998 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.734456 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.735238 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.735478 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.735561 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.735753 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:08:21.735784 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:08:21.735842 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:21.735894 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 19:08:21.735914 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:08:21.736028 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.733527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:35531 (local address 127.0.203.2:53775)
0620 19:08:21.733698 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:21.733711 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:21.733774 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:21.733897 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:21.733898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:21.733899 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:08:21.733944 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.733945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.734535 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.734536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.735172 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.735175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.735581 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.735582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.735649 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.735661 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:08:21.735848 (+ 187us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:08:21.735849 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 19:08:21.735893 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 19:08:21.735939 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260620 19:08:21.738418 8747 raft_consensus.cc:1261] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 8b9ddaf367644bacb3aac674f33582c1 [term 1 FOLLOWER]: Refusing update from remote peer 2073bd7db4d14d9b8bd9ea0c5cb65696: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:21.738497 8465 raft_consensus.cc:1261] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P b869b8b864be40fa9446f0e98683c84f [term 1 FOLLOWER]: Refusing update from remote peer 2073bd7db4d14d9b8bd9ea0c5cb65696: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:21.738659 8829 consensus_queue.cc:1035] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9ddaf367644bacb3aac674f33582c1" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:21.738801 8903 consensus_queue.cc:1035] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b869b8b864be40fa9446f0e98683c84f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 37653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:21.746730 8820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.744224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:60433 (local address 127.0.203.2:40787)
0620 19:08:21.744259 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:21.744262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:21.744284 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:08:21.744305 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:21.744305 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:21.744315 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:21.744371 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:08:21.744524 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.745236 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.746016 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.746287 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.746368 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.746541 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:08:21.746567 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:08:21.746608 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:21.746666 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 19:08:21.746683 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:08:21.746785 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:21.744138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:40787 (local address 127.0.203.3:60433)
0620 19:08:21.744186 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:21.744198 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:21.744238 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:21.744414 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:21.744415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:21.744416 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:08:21.744454 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.744456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.745331 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.745332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.745921 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:21.745923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:21.746404 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:21.746405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:21.746462 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:21.746469 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:08:21.746611 (+ 142us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:08:21.746613 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:08:21.746664 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:08:21.746714 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 19:08:21.753875 8309 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:21.753997 8309 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:21.754575 8309 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed57b61f7c3d4b33b70d661eeee2f354: Sending DeleteTablet for 3 replicas of tablet 1a20efa5ee4f48158e5bd2a593f6b0d1
I20260620 19:08:21.754812 8727 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a20efa5ee4f48158e5bd2a593f6b0d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:21 UTC) from {username='oryx', principal='oryx/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:46452
I20260620 19:08:21.754904 8847 tablet_replica.cc:330] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 8b9ddaf367644bacb3aac674f33582c1: stopping tablet replica
I20260620 19:08:21.754889 8446 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a20efa5ee4f48158e5bd2a593f6b0d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:21 UTC) from {username='oryx', principal='oryx/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:37306
I20260620 19:08:21.754943 8847 raft_consensus.cc:2229] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 8b9ddaf367644bacb3aac674f33582c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:21.754977 8847 raft_consensus.cc:2258] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 8b9ddaf367644bacb3aac674f33582c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:21.754977 8846 tablet_replica.cc:330] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P b869b8b864be40fa9446f0e98683c84f: stopping tablet replica
I20260620 19:08:21.755019 8846 raft_consensus.cc:2229] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P b869b8b864be40fa9446f0e98683c84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:21.755056 8846 raft_consensus.cc:2258] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P b869b8b864be40fa9446f0e98683c84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:21.755339 8587 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a20efa5ee4f48158e5bd2a593f6b0d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:21 UTC) from {username='oryx', principal='oryx/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:60452
I20260620 19:08:21.755424 8845 tablet_replica.cc:330] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696: stopping tablet replica
I20260620 19:08:21.755472 8845 raft_consensus.cc:2229] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:08:21.755551 8845 raft_consensus.cc:2258] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:21.755791 8847 ts_tablet_manager.cc:1905] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 8b9ddaf367644bacb3aac674f33582c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:21.757265 8845 ts_tablet_manager.cc:1905] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:21.758370 8845 ts_tablet_manager.cc:1918] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:08:21.758452 8845 log.cc:1196] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/1a20efa5ee4f48158e5bd2a593f6b0d1
I20260620 19:08:21.758661 8845 ts_tablet_manager.cc:1939] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 2073bd7db4d14d9b8bd9ea0c5cb65696: Deleting consensus metadata
I20260620 19:08:21.759115 8296 catalog_manager.cc:4841] TS 2073bd7db4d14d9b8bd9ea0c5cb65696 (127.0.203.2:40787): tablet 1a20efa5ee4f48158e5bd2a593f6b0d1 (table test-table [id=e24992f80b3a4f08908bf3a18dadd94c]) successfully deleted
I20260620 19:08:21.759354 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 8382
I20260620 19:08:21.763109 8847 ts_tablet_manager.cc:1918] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 8b9ddaf367644bacb3aac674f33582c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:08:21.763211 8847 log.cc:1196] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 8b9ddaf367644bacb3aac674f33582c1: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/1a20efa5ee4f48158e5bd2a593f6b0d1
I20260620 19:08:21.763425 8847 ts_tablet_manager.cc:1939] T 1a20efa5ee4f48158e5bd2a593f6b0d1 P 8b9ddaf367644bacb3aac674f33582c1: Deleting consensus metadata
I20260620 19:08:21.763749 8294 catalog_manager.cc:4841] TS 8b9ddaf367644bacb3aac674f33582c1 (127.0.203.3:35531): tablet 1a20efa5ee4f48158e5bd2a593f6b0d1 (table test-table [id=e24992f80b3a4f08908bf3a18dadd94c]) successfully deleted
W20260620 19:08:21.767802 8295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.1:37653 (error 108)
W20260620 19:08:21.768500 8913 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:21.768215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:37653 (local address 127.0.0.1:37328)
0620 19:08:21.768333 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:21.768345 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:21.768395 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:21.768442 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.1:37653: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":71,"threads_started":1}
I20260620 19:08:21.768712 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 8522
W20260620 19:08:21.768724 8295 catalog_manager.cc:4584] TS b869b8b864be40fa9446f0e98683c84f (127.0.203.1:37653): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a20efa5ee4f48158e5bd2a593f6b0d1: Network error: Client connection negotiation failed: client connection to 127.0.203.1:37653: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260620 19:08:21.774677 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 8662
I20260620 19:08:21.779960 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 8278
2026-06-20T19:08:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2683 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:08:21 dist-test-slave-wd30 krb5kdc[8916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:08:21 dist-test-slave-wd30 krb5kdc[8916](info): set up 2 sockets
Jun 20 19:08:21 dist-test-slave-wd30 krb5kdc[8916](info): commencing operation
krb5kdc: starting...
W20260620 19:08:22.843756 812 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:08:22 dist-test-slave-wd30 krb5kdc[8916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:08:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:08:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:08:22.960179 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:42661
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:34875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:42661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:23.041265 8929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:23.041476 8929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:23.041497 8929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:23.042671 8929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:23.042714 8929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:23.042729 8929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:23.042742 8929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:23.042754 8929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:08:23.044402 8929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:34875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:42661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:42661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:23.044669 8929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:23.044910 8929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:23.049826 8935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:23.049862 8934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:23.049827 8937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:23.050071 8929 server_base.cc:1034] running on GCE node
I20260620 19:08:23.050311 8929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:23.050643 8929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:23.051807 8929 hybrid_clock.cc:648] HybridClock initialized: now 1781982503051797 us; error 40 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T19:08:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1276 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:08:23 dist-test-slave-wd30 krb5kdc[8942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:08:23 dist-test-slave-wd30 krb5kdc[8942](info): set up 2 sockets
Jun 20 19:08:23 dist-test-slave-wd30 krb5kdc[8942](info): commencing operation
krb5kdc: starting...
W20260620 19:08:24.091761 812 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:08:24 dist-test-slave-wd30 krb5kdc[8942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:08:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:08:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:08:24.206494 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:43469
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:38687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:43469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:24.286806 8955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:24.287062 8955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:24.287101 8955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:24.288293 8955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:24.288370 8955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:24.288393 8955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:24.288414 8955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:24.288444 8955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:08:24.290016 8955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:38687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:43469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:43469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:24.290292 8955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:24.290532 8955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:24.295408 8963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:24.295413 8960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:24.295450 8961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:24.295579 8955 server_base.cc:1034] running on GCE node
I20260620 19:08:24.295900 8955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:24.296200 8955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:24.297349 8955 hybrid_clock.cc:648] HybridClock initialized: now 1781982504297332 us; error 35 us; skew 500 ppm
Jun 20 19:08:24 dist-test-slave-wd30 krb5kdc[8942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:24.299907 8955 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:08:24.300381 8955 webserver.cc:460] Webserver started at http://127.0.203.62:42827/ using document root <none> and password file <none>
I20260620 19:08:24.300617 8955 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:24.300685 8955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:24.300843 8955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:24.301916 8955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "60ce037b460f4d298cc41ce391e88936"
format_stamp: "Formatted at 2026-06-20 19:08:24 on dist-test-slave-wd30"
server_key: "4b80e51eea695d0110d692894f671102"
server_key_iv: "b88982d85cb84ba42090a2cd0f07dfb4"
server_key_version: "encryptionkey@0"
I20260620 19:08:24.302279 8955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "60ce037b460f4d298cc41ce391e88936"
format_stamp: "Formatted at 2026-06-20 19:08:24 on dist-test-slave-wd30"
server_key: "4b80e51eea695d0110d692894f671102"
server_key_iv: "b88982d85cb84ba42090a2cd0f07dfb4"
server_key_version: "encryptionkey@0"
I20260620 19:08:24.303674 8955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:08:24.304365 8970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:24.304522 8955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:24.304636 8955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "60ce037b460f4d298cc41ce391e88936"
format_stamp: "Formatted at 2026-06-20 19:08:24 on dist-test-slave-wd30"
server_key: "4b80e51eea695d0110d692894f671102"
server_key_iv: "b88982d85cb84ba42090a2cd0f07dfb4"
server_key_version: "encryptionkey@0"
I20260620 19:08:24.304711 8955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:24.357334 8955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:24.359436 8955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:24.359601 8955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:24.363590 8955 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:43469
I20260620 19:08:24.363642 9022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:43469 every 8 connection(s)
I20260620 19:08:24.364140 8955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:24.364784 9023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:24.366972 9023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936: Bootstrap starting.
I20260620 19:08:24.367581 9023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:24.367830 9023 log.cc:824] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:24.368456 9023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936: No bootstrap required, opened a new log
I20260620 19:08:24.369971 9023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ce037b460f4d298cc41ce391e88936" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 43469 } }
I20260620 19:08:24.370105 9023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:24.370127 9023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ce037b460f4d298cc41ce391e88936, State: Initialized, Role: FOLLOWER
I20260620 19:08:24.370271 9023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ce037b460f4d298cc41ce391e88936" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 43469 } }
I20260620 19:08:24.370355 9023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:24.370379 9023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:24.370414 9023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:24.371307 9023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ce037b460f4d298cc41ce391e88936" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 43469 } }
I20260620 19:08:24.371394 9023 leader_election.cc:302] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60ce037b460f4d298cc41ce391e88936; no voters:
I20260620 19:08:24.371515 9023 leader_election.cc:288] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:08:24.371588 9026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:24.371773 9023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:24.371784 9026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [term 1 LEADER]: Becoming Leader. State: Replica: 60ce037b460f4d298cc41ce391e88936, State: Running, Role: LEADER
I20260620 19:08:24.371984 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 8955
I20260620 19:08:24.372030 9026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ce037b460f4d298cc41ce391e88936" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 43469 } }
I20260620 19:08:24.372088 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:08:24.372277 812 external_mini_cluster.cc:1428] Setting key 61aacf34c043772b3afcb8a3654d3b28
I20260620 19:08:24.372416 9028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60ce037b460f4d298cc41ce391e88936. Latest consensus state: current_term: 1 leader_uuid: "60ce037b460f4d298cc41ce391e88936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ce037b460f4d298cc41ce391e88936" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 43469 } } }
I20260620 19:08:24.372491 9028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:24.372727 9027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60ce037b460f4d298cc41ce391e88936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ce037b460f4d298cc41ce391e88936" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 43469 } } }
I20260620 19:08:24.373111 9027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:24.373272 9035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:08:24.373971 9035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:08:24.376344 9035 catalog_manager.cc:1347] Generated new cluster ID: d50f50f97e44466684144520ce09b0eb
I20260620 19:08:24.376397 9035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:08:24 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:24.383884 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:24.372907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38854 (local address 127.0.203.62:43469)
0620 19:08:24.373105 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:24.373114 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:24.373384 (+ 270us) server_negotiation.cc:407] Connection header received
0620 19:08:24.373504 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:24.373512 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:24.373567 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:24.373682 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:24.374505 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:24.375880 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:24.376881 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:24.377534 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:24.377646 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:24.380416 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:24.380420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:24.380420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:24.380431 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:24.382279 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:24.382675 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:24.382678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:24.382678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:24.382782 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:24.382991 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:24.382991 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:24.382992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:24.383292 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:24.383445 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:24.383555 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 19:08:24.383680 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":61,"threads_started":1}
I20260620 19:08:24.389818 9035 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:08:24.390355 9035 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:24.395655 9035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936: Generated new TSK 0
I20260620 19:08:24.395862 9035 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
I20260620 19:08:24.421957 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:43469
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:38687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:24.514331 9049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:24.514542 9049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:24.514567 9049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:24.515839 9049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:24.515883 9049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:24.515911 9049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:08:24.517825 9049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:38687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:43469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:24.518090 9049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:24.518340 9049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:08:24.524137 9049 server_base.cc:1034] running on GCE node
W20260620 19:08:24.524063 9055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:24.524161 9057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:24.524063 9054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:24.524482 9049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:24.524683 9049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:24.525851 9049 hybrid_clock.cc:648] HybridClock initialized: now 1781982504525836 us; error 32 us; skew 500 ppm
Jun 20 19:08:24 dist-test-slave-wd30 krb5kdc[8942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:24.528497 9049 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:08:24.528935 9049 webserver.cc:460] Webserver started at http://127.0.203.1:35913/ using document root <none> and password file <none>
I20260620 19:08:24.529165 9049 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:24.529234 9049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:24.529392 9049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:24.530449 9049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "e960ec74a5994911bd2e3f6905ba99c5"
format_stamp: "Formatted at 2026-06-20 19:08:24 on dist-test-slave-wd30"
server_key: "c9555cc8cb8d68880bb1c8de9286c709"
server_key_iv: "e85ffc7da332cca908937696e8d664c1"
server_key_version: "encryptionkey@0"
I20260620 19:08:24.530813 9049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "e960ec74a5994911bd2e3f6905ba99c5"
format_stamp: "Formatted at 2026-06-20 19:08:24 on dist-test-slave-wd30"
server_key: "c9555cc8cb8d68880bb1c8de9286c709"
server_key_iv: "e85ffc7da332cca908937696e8d664c1"
server_key_version: "encryptionkey@0"
I20260620 19:08:24.532193 9049 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:08:24.532917 9064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:24.533080 9049 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:24.533181 9049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "e960ec74a5994911bd2e3f6905ba99c5"
format_stamp: "Formatted at 2026-06-20 19:08:24 on dist-test-slave-wd30"
server_key: "c9555cc8cb8d68880bb1c8de9286c709"
server_key_iv: "e85ffc7da332cca908937696e8d664c1"
server_key_version: "encryptionkey@0"
I20260620 19:08:24.533265 9049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:24.544385 9049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:24.546096 9049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:24.546262 9049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:24.546478 9049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:24.546779 9049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:24.546833 9049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:24.546875 9049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:24.546903 9049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:24.552678 9049 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:34163
I20260620 19:08:24.552716 9177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:34163 every 8 connection(s)
I20260620 19:08:24.553020 9049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:08:24.557169 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 9049
I20260620 19:08:24.557294 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:08:24.557497 812 external_mini_cluster.cc:1428] Setting key e37f76e2e1a742a2219be2f4b8aced23
Jun 20 19:08:24 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:24.562642 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:24.553760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:45809 (local address 127.0.203.62:43469)
0620 19:08:24.553824 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:24.553826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:24.554039 (+ 213us) server_negotiation.cc:407] Connection header received
0620 19:08:24.554946 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:24.554947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:24.554959 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:24.555015 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:24.555608 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:24.556105 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:24.557128 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:24.557407 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:24.557472 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:24.559846 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:24.559847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:24.559848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:24.559855 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:24.561182 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:24.561680 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:24.561681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:24.561681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:24.561740 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:24.562027 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:24.562027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:24.562028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:24.562124 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:24.562172 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:24.562510 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 19:08:24.562578 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:08:24.562965 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:24.553735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.1:45809)
0620 19:08:24.553913 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:24.553953 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:24.554860 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:24.555069 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:24.555073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:24.555339 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:24.555520 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:24.555524 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:24.556184 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:24.556185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:24.557023 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:24.557025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:24.557494 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:24.557495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:24.557651 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:24.558203 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:24.558211 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:24.559730 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:24.561267 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:24.561274 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:24.561277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:24.561604 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:24.561812 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:24.561813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:24.561814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:24.561960 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:24.562190 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:24.562196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:24.562363 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 19:08:24.562491 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 19:08:24.562670 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":111,"threads_started":1}
I20260620 19:08:24.563534 9178 heartbeater.cc:347] Connected to a master server at 127.0.203.62:43469
I20260620 19:08:24.563627 9178 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:24.563815 9178 heartbeater.cc:511] Master 127.0.203.62:43469 requested a full tablet report, sending...
I20260620 19:08:24.564292 8987 ts_manager.cc:194] Registered new tserver with Master: e960ec74a5994911bd2e3f6905ba99c5 (127.0.203.1:34163)
I20260620 19:08:24.564883 8987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:45809
WARNING: no policy specified for kudu/127.0.203.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
I20260620 19:08:24.581897 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:43469
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:38687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:24.662576 9183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:24.662822 9183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:24.662863 9183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:24.664091 9183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:24.664157 9183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:24.664203 9183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:08:24.665706 9183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:38687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:43469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:24.665975 9183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:24.666205 9183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:24.671039 9189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:24.671065 9191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:24.671029 9188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:24.671198 9183 server_base.cc:1034] running on GCE node
I20260620 19:08:24.671643 9183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:24.671903 9183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:24.673040 9183 hybrid_clock.cc:648] HybridClock initialized: now 1781982504673018 us; error 36 us; skew 500 ppm
Jun 20 19:08:24 dist-test-slave-wd30 krb5kdc[8942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:24.675452 9183 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:08:24.675889 9183 webserver.cc:460] Webserver started at http://127.0.203.2:35209/ using document root <none> and password file <none>
I20260620 19:08:24.676136 9183 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:24.676203 9183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:24.676343 9183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:24.677371 9183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "4a6fe7e6c56b4c22b8958922dd971262"
format_stamp: "Formatted at 2026-06-20 19:08:24 on dist-test-slave-wd30"
server_key: "6b0c89f07b6e1604685f6261f06e86a8"
server_key_iv: "e754ffb579662eb2896dd72ccf4cc295"
server_key_version: "encryptionkey@0"
I20260620 19:08:24.677765 9183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "4a6fe7e6c56b4c22b8958922dd971262"
format_stamp: "Formatted at 2026-06-20 19:08:24 on dist-test-slave-wd30"
server_key: "6b0c89f07b6e1604685f6261f06e86a8"
server_key_iv: "e754ffb579662eb2896dd72ccf4cc295"
server_key_version: "encryptionkey@0"
I20260620 19:08:24.679168 9183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:08:24.679950 9198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:24.680132 9183 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:08:24.680228 9183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "4a6fe7e6c56b4c22b8958922dd971262"
format_stamp: "Formatted at 2026-06-20 19:08:24 on dist-test-slave-wd30"
server_key: "6b0c89f07b6e1604685f6261f06e86a8"
server_key_iv: "e754ffb579662eb2896dd72ccf4cc295"
server_key_version: "encryptionkey@0"
I20260620 19:08:24.680306 9183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:24.694041 9183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:24.695940 9183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:24.696103 9183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:24.696331 9183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:24.696646 9183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:24.696699 9183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:24.696751 9183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:24.696779 9183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:24.702564 9183 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:38115
I20260620 19:08:24.702608 9311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:38115 every 8 connection(s)
I20260620 19:08:24.702922 9183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:08:24.706538 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 9183
I20260620 19:08:24.706640 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:08:24.706845 812 external_mini_cluster.cc:1428] Setting key 4126a3da51443c2e4275484bda44ac82
Jun 20 19:08:24 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:24.713001 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:24.703690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:42099 (local address 127.0.203.62:43469)
0620 19:08:24.703738 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:24.703740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:24.703939 (+ 199us) server_negotiation.cc:407] Connection header received
0620 19:08:24.704859 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:24.704860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:24.704869 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:24.704931 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:24.705518 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:24.706016 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:24.707389 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:24.707630 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:24.707726 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:24.709984 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:24.709986 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:24.709987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:24.709995 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:24.711527 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:24.712021 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:24.712021 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:24.712022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:24.712073 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:24.712366 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:24.712366 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:24.712366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:24.712475 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:24.712524 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:24.712870 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 19:08:24.712936 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:08:24.713342 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:24.703644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.2:42099)
0620 19:08:24.703807 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:24.703849 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:24.704769 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:24.704984 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:24.704988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:24.705244 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:24.705434 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:24.705438 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:24.706126 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:24.706127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:24.707314 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:24.707317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:24.707739 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:24.707741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:24.707885 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:24.708496 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:24.708504 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:24.709881 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:24.711612 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:24.711618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:24.711621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:24.711939 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:24.712137 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:24.712138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:24.712138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:24.712296 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:24.712540 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:24.712547 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:24.712724 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 19:08:24.712856 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 19:08:24.713044 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260620 19:08:24.713861 9312 heartbeater.cc:347] Connected to a master server at 127.0.203.62:43469
I20260620 19:08:24.713984 9312 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:24.714178 9312 heartbeater.cc:511] Master 127.0.203.62:43469 requested a full tablet report, sending...
I20260620 19:08:24.714603 8987 ts_manager.cc:194] Registered new tserver with Master: 4a6fe7e6c56b4c22b8958922dd971262 (127.0.203.2:38115)
I20260620 19:08:24.714972 8987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:42099
WARNING: no policy specified for kudu/127.0.203.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
I20260620 19:08:24.733819 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:43469
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:38687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:24.815217 9317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:24.815408 9317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:24.815430 9317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:24.816537 9317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:24.816570 9317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:24.816596 9317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:08:24.818082 9317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:38687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:43469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:24.818267 9317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:24.818435 9317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:24.823398 9325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:24.823429 9317 server_base.cc:1034] running on GCE node
W20260620 19:08:24.823465 9323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:24.823398 9322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:24.823822 9317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:24.824105 9317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:24.825259 9317 hybrid_clock.cc:648] HybridClock initialized: now 1781982504825248 us; error 28 us; skew 500 ppm
Jun 20 19:08:24 dist-test-slave-wd30 krb5kdc[8942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:24.827839 9317 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:08:24.828310 9317 webserver.cc:460] Webserver started at http://127.0.203.3:33161/ using document root <none> and password file <none>
I20260620 19:08:24.828534 9317 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:24.828578 9317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:24.828715 9317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:24.829741 9317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "135625f6d4dd41b8ae035cb488fefc52"
format_stamp: "Formatted at 2026-06-20 19:08:24 on dist-test-slave-wd30"
server_key: "76b636879277200adb5eff7d9bb36569"
server_key_iv: "8f69ced441d82ba648d10b0a78403553"
server_key_version: "encryptionkey@0"
I20260620 19:08:24.830111 9317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "135625f6d4dd41b8ae035cb488fefc52"
format_stamp: "Formatted at 2026-06-20 19:08:24 on dist-test-slave-wd30"
server_key: "76b636879277200adb5eff7d9bb36569"
server_key_iv: "8f69ced441d82ba648d10b0a78403553"
server_key_version: "encryptionkey@0"
I20260620 19:08:24.831383 9317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:08:24.832057 9332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:24.832221 9317 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:08:24.832309 9317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "135625f6d4dd41b8ae035cb488fefc52"
format_stamp: "Formatted at 2026-06-20 19:08:24 on dist-test-slave-wd30"
server_key: "76b636879277200adb5eff7d9bb36569"
server_key_iv: "8f69ced441d82ba648d10b0a78403553"
server_key_version: "encryptionkey@0"
I20260620 19:08:24.832363 9317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:24.856550 9317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:24.858908 9317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:24.859023 9317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:24.859239 9317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:24.859498 9317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:24.859529 9317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:24.859552 9317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:24.859565 9317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:24.865017 9317 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:42961
I20260620 19:08:24.865096 9445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:42961 every 8 connection(s)
I20260620 19:08:24.865417 9317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:08:24.868636 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 9317
I20260620 19:08:24.868733 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:08:24.868930 812 external_mini_cluster.cc:1428] Setting key 5c9c1cadb85d0a20f174d557b1994f43
Jun 20 19:08:24 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:24.875355 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:24.866175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:57881 (local address 127.0.203.62:43469)
0620 19:08:24.866269 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:24.866271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:24.866487 (+ 216us) server_negotiation.cc:407] Connection header received
0620 19:08:24.867501 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:24.867502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:24.867514 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:24.867593 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:24.868263 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:24.868868 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:24.870181 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:24.870374 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:24.870431 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:24.872755 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:24.872756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:24.872757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:24.872763 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:24.874183 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:24.874543 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:24.874544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:24.874544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:24.874594 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:24.874795 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:24.874796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:24.874796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:24.874894 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:24.874944 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:24.875229 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 19:08:24.875290 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 19:08:24.875540 9448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:24.866165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.3:57881)
0620 19:08:24.866367 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:24.866402 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:24.867383 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:24.867692 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:24.867696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:24.867971 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:24.868160 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:24.868164 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:24.868954 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:24.868955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:24.870110 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:24.870112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:24.870444 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:24.870445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:24.870573 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:24.871105 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:24.871119 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:24.872674 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:24.874260 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:24.874266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:24.874267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:24.874477 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:24.874650 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:24.874650 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:24.874651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:24.874742 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:24.874947 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:24.874950 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:24.875062 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 19:08:24.875203 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 19:08:24.875330 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":123,"threads_started":1}
I20260620 19:08:24.876050 9446 heartbeater.cc:347] Connected to a master server at 127.0.203.62:43469
I20260620 19:08:24.876156 9446 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:24.876331 9446 heartbeater.cc:511] Master 127.0.203.62:43469 requested a full tablet report, sending...
I20260620 19:08:24.876663 8987 ts_manager.cc:194] Registered new tserver with Master: 135625f6d4dd41b8ae035cb488fefc52 (127.0.203.3:42961)
I20260620 19:08:24.877068 8987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:57881
I20260620 19:08:24.880800 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/oryx_127.0.203.1.keytab.
Entry for principal oryx/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/oryx_127.0.203.1.keytab.
WARNING: no policy specified for oryx/127.0.203.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/oryx_127.0.203.2.keytab.
Entry for principal oryx/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/oryx_127.0.203.2.keytab.
WARNING: no policy specified for oryx/127.0.203.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/oryx_127.0.203.3.keytab.
Entry for principal oryx/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/oryx_127.0.203.3.keytab.
I20260620 19:08:24.949258 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 9049
I20260620 19:08:24.954116 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 9183
I20260620 19:08:24.958485 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 9317
I20260620 19:08:24.962518 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 8955
I20260620 19:08:24.967705 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:43469
--webserver_interface=127.0.203.62
--webserver_port=42827
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:38687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:43469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:25.047333 9455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:25.047533 9455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:25.047555 9455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:25.048655 9455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:25.048691 9455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:25.048704 9455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:25.048717 9455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:25.048729 9455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:08:25.050168 9455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:38687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:43469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:43469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=42827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:25.050361 9455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:25.050524 9455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:25.055332 9460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:25.055323 9463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:25.055341 9461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:25.055451 9455 server_base.cc:1034] running on GCE node
I20260620 19:08:25.055733 9455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:25.055974 9455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:25.057111 9455 hybrid_clock.cc:648] HybridClock initialized: now 1781982505057093 us; error 33 us; skew 500 ppm
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:25.059659 9455 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:08:25.060070 9455 webserver.cc:460] Webserver started at http://127.0.203.62:42827/ using document root <none> and password file <none>
I20260620 19:08:25.060297 9455 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:25.060379 9455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:25.061750 9455 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:25.062422 9470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:25.062616 9455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 19:08:25.062707 9455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "60ce037b460f4d298cc41ce391e88936"
format_stamp: "Formatted at 2026-06-20 19:08:24 on dist-test-slave-wd30"
server_key: "4b80e51eea695d0110d692894f671102"
server_key_iv: "b88982d85cb84ba42090a2cd0f07dfb4"
server_key_version: "encryptionkey@0"
I20260620 19:08:25.063038 9455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:25.080519 9455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:25.082828 9455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:25.082998 9455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:25.086800 9455 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:43469
I20260620 19:08:25.086853 9522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:43469 every 8 connection(s)
I20260620 19:08:25.087378 9455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:25.088614 9523 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:08:25.089247 9523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936: Bootstrap starting.
I20260620 19:08:25.092555 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 9455
I20260620 19:08:25.092892 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:34163
--local_ip_for_outbound_sockets=127.0.203.1
--tserver_master_addrs=127.0.203.62:43469
--webserver_port=35913
--webserver_interface=127.0.203.1
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:38687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.203.1
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/oryx_127.0.203.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:08:25.097891 9523 log.cc:824] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:25.099669 9523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:08:25.099963 9523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936: Bootstrap complete.
I20260620 19:08:25.102200 9523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ce037b460f4d298cc41ce391e88936" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 43469 } }
I20260620 19:08:25.102450 9523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ce037b460f4d298cc41ce391e88936, State: Initialized, Role: FOLLOWER
I20260620 19:08:25.102654 9523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ce037b460f4d298cc41ce391e88936" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 43469 } }
I20260620 19:08:25.102756 9523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:25.102809 9523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:25.102859 9523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:08:25.103930 9523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ce037b460f4d298cc41ce391e88936" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 43469 } }
I20260620 19:08:25.104049 9523 leader_election.cc:302] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60ce037b460f4d298cc41ce391e88936; no voters:
I20260620 19:08:25.104254 9523 leader_election.cc:288] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:08:25.104291 9527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:08:25.104403 9527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [term 2 LEADER]: Becoming Leader. State: Replica: 60ce037b460f4d298cc41ce391e88936, State: Running, Role: LEADER
I20260620 19:08:25.104526 9523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:25.104554 9527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ce037b460f4d298cc41ce391e88936" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 43469 } }
I20260620 19:08:25.104938 9528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "60ce037b460f4d298cc41ce391e88936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ce037b460f4d298cc41ce391e88936" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 43469 } } }
I20260620 19:08:25.105020 9528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:25.105439 9529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60ce037b460f4d298cc41ce391e88936. Latest consensus state: current_term: 2 leader_uuid: "60ce037b460f4d298cc41ce391e88936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ce037b460f4d298cc41ce391e88936" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 43469 } } }
I20260620 19:08:25.105513 9529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936 [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:25.105664 9533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:08:25.106142 9533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:08:25.106410 9533 catalog_manager.cc:1259] Loaded cluster ID: d50f50f97e44466684144520ce09b0eb
I20260620 19:08:25.106514 9533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:08:25.107697 9533 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:25.107872 9533 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 60ce037b460f4d298cc41ce391e88936: Loaded TSK: 0
I20260620 19:08:25.108084 9533 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:08:25.174667 9525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:25.174904 9525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:25.174952 9525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:25.176156 9525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:25.176219 9525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:25.176265 9525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:08:25.177747 9525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:38687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/oryx_127.0.203.1.keytab
--principal=oryx/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:34163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=35913
--tserver_master_addrs=127.0.203.62:43469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:25.177997 9525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:25.178236 9525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:25.183983 9552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:25.183995 9549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:25.183991 9550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:25.184190 9525 server_base.cc:1034] running on GCE node
I20260620 19:08:25.184476 9525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:25.184756 9525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:25.185909 9525 hybrid_clock.cc:648] HybridClock initialized: now 1781982505185898 us; error 26 us; skew 500 ppm
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982505, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:25.188572 9525 init.cc:377] Logged in from keytab as oryx/127.0.203.1@KRBTEST.COM (short username oryx)
I20260620 19:08:25.189095 9525 webserver.cc:460] Webserver started at http://127.0.203.1:35913/ using document root <none> and password file <none>
I20260620 19:08:25.189366 9525 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:25.189450 9525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:25.191200 9525 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:08:25.191875 9559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:25.192086 9525 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:25.192178 9525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "e960ec74a5994911bd2e3f6905ba99c5"
format_stamp: "Formatted at 2026-06-20 19:08:24 on dist-test-slave-wd30"
server_key: "c9555cc8cb8d68880bb1c8de9286c709"
server_key_iv: "e85ffc7da332cca908937696e8d664c1"
server_key_version: "encryptionkey@0"
I20260620 19:08:25.192622 9525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:25.210865 9525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:25.213228 9525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:25.213405 9525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:25.213670 9525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:25.214035 9525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:25.214087 9525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:25.214120 9525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:25.214145 9525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:25.221328 9525 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:34163
I20260620 19:08:25.221398 9672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:34163 every 8 connection(s)
I20260620 19:08:25.221768 9525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:08:25.228152 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 9525
I20260620 19:08:25.228626 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:38115
--local_ip_for_outbound_sockets=127.0.203.2
--tserver_master_addrs=127.0.203.62:43469
--webserver_port=35209
--webserver_interface=127.0.203.2
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:38687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.203.2
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/oryx_127.0.203.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:25.231168 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:25.222471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.1:60799)
0620 19:08:25.222625 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:25.222667 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:25.223234 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:25.223691 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:25.223696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:25.224292 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:25.224484 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:25.224489 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.225647 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.225648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:25.226501 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:25.226504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.227111 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.227112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:25.227279 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:25.227895 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:25.227922 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:25.230914 (+ 2992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
W20260620 19:08:25.231429 9673 heartbeater.cc:644] Failed to heartbeat to 127.0.203.62:43469 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.203.62:43469: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
W20260620 19:08:25.231760 9676 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:25.222568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:60799 (local address 127.0.203.62:43469)
0620 19:08:25.222761 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:25.222769 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:25.222809 (+ 40us) server_negotiation.cc:407] Connection header received
0620 19:08:25.223453 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:25.223458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:25.223504 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:25.223650 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:25.224601 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.225535 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.226593 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.227015 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.227162 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:25.231059 (+ 3897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.1:60799: BlockingRecv error: recv got EOF from 127.0.203.1:60799 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:25.237780 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:25.231649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.1:48269)
0620 19:08:25.231955 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:25.231965 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:25.232021 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:25.232187 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:25.232188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:25.232354 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:25.232413 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:25.232415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.233420 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.233421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:25.234367 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:25.234370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.234944 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.234945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:25.235019 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:25.235557 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:25.235562 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:25.237687 (+ 2125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291}
W20260620 19:08:25.238186 9676 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:25.231716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:48269 (local address 127.0.203.62:43469)
0620 19:08:25.231926 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:25.231928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:25.232070 (+ 142us) server_negotiation.cc:407] Connection header received
0620 19:08:25.232093 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:25.232094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:25.232108 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:25.232150 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:25.232503 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.233318 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.234468 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.234810 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.234886 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:25.238127 (+ 3241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.1:48269: BlockingRecv error: recv got EOF from 127.0.203.1:48269 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":108,"threads_started":1}
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:25.243201 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:25.238050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.1:41207)
0620 19:08:25.238285 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:25.238297 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:25.238338 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:25.238523 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:25.238524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:25.238683 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:25.238720 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:25.238721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.239600 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.239601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:25.240489 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:25.240492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.240941 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.240942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:25.240996 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:25.241424 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:25.241430 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:25.243135 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":99,"threads_started":1}
W20260620 19:08:25.243307 9673 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:08:25.243438 9676 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:25.238239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:41207 (local address 127.0.203.62:43469)
0620 19:08:25.238396 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:25.238398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:25.238406 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:08:25.238427 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:25.238428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:25.238437 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:25.238498 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:25.238800 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.239518 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.240572 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.240869 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.240931 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:25.243222 (+ 2291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.1:41207: BlockingRecv error: recv got EOF from 127.0.203.1:41207 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
W20260620 19:08:25.313444 9677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:25.313673 9677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:25.313710 9677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:25.314878 9677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:25.314942 9677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:25.314987 9677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:08:25.316459 9677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:38687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/oryx_127.0.203.2.keytab
--principal=oryx/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:38115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=35209
--tserver_master_addrs=127.0.203.62:43469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:25.316721 9677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:25.316962 9677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:25.321913 9687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:25.321923 9685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:25.321956 9677 server_base.cc:1034] running on GCE node
W20260620 19:08:25.321936 9684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:25.322420 9677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:25.322659 9677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:25.323817 9677 hybrid_clock.cc:648] HybridClock initialized: now 1781982505323817 us; error 41 us; skew 500 ppm
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982505, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:25.326229 9677 init.cc:377] Logged in from keytab as oryx/127.0.203.2@KRBTEST.COM (short username oryx)
I20260620 19:08:25.326663 9677 webserver.cc:460] Webserver started at http://127.0.203.2:35209/ using document root <none> and password file <none>
I20260620 19:08:25.326839 9677 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:25.326901 9677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:25.328497 9677 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:25.329131 9694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:25.329314 9677 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:25.329412 9677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "4a6fe7e6c56b4c22b8958922dd971262"
format_stamp: "Formatted at 2026-06-20 19:08:24 on dist-test-slave-wd30"
server_key: "6b0c89f07b6e1604685f6261f06e86a8"
server_key_iv: "e754ffb579662eb2896dd72ccf4cc295"
server_key_version: "encryptionkey@0"
I20260620 19:08:25.329762 9677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:25.342895 9677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:25.344847 9677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:25.345007 9677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:25.345270 9677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:25.345575 9677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:25.345630 9677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:25.345674 9677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:25.345697 9677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:25.351691 9677 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:38115
I20260620 19:08:25.351725 9807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:38115 every 8 connection(s)
I20260620 19:08:25.352066 9677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:08:25.354614 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 9677
I20260620 19:08:25.354894 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:42961
--local_ip_for_outbound_sockets=127.0.203.3
--tserver_master_addrs=127.0.203.62:43469
--webserver_port=33161
--webserver_interface=127.0.203.3
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:38687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.203.3
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/oryx_127.0.203.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:08:25.360773 9676 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:25.352756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:59169 (local address 127.0.203.62:43469)
0620 19:08:25.352821 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:25.352823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:25.353142 (+ 319us) server_negotiation.cc:407] Connection header received
0620 19:08:25.353677 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:25.353679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:25.353696 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:25.353741 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:25.354692 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.355387 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.357016 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.357333 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.357374 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:25.360724 (+ 3350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.2:59169: BlockingRecv error: recv got EOF from 127.0.203.2:59169 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:08:25.360976 9810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:25.352745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.2:59169)
0620 19:08:25.352927 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:25.352961 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:25.353561 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:25.353824 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:25.353828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:25.354383 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:25.354588 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:25.354592 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.355504 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.355505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:25.356907 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:25.356911 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.357464 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.357465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:25.357625 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:25.358194 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:25.358202 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:25.360646 (+ 2444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
W20260620 19:08:25.361225 9808 heartbeater.cc:644] Failed to heartbeat to 127.0.203.62:43469 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.203.62:43469: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:25.366353 9810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:25.361502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.2:37051)
0620 19:08:25.361552 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:25.361562 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:25.361601 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:25.361727 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:25.361727 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:25.361891 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:25.361950 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:25.361952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.362532 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.362533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:25.363468 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:25.363471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.363788 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.363789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:25.363863 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:25.364240 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:25.364246 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:25.366248 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 19:08:25.366427 9676 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:25.361548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:37051 (local address 127.0.203.62:43469)
0620 19:08:25.361582 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:25.361584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:25.361592 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:08:25.361664 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:25.361664 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:25.361671 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:25.361715 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:25.362020 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.362457 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.363552 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.363719 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.363769 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:25.366351 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.2:37051: BlockingRecv error: recv got EOF from 127.0.203.2:37051 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:25.370956 9810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:25.366826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.2:60329)
0620 19:08:25.366874 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:25.366884 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:25.366921 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:25.367032 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:25.367033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:25.367182 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:25.367213 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:25.367214 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.367773 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.367774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:25.368570 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:25.368572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.368868 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.368869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:25.368922 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:25.369269 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:25.369274 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:25.370892 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 19:08:25.371001 9676 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:25.366869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:60329 (local address 127.0.203.62:43469)
0620 19:08:25.366918 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:25.366920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:25.366933 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:08:25.366971 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:25.366972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:25.366979 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:25.367019 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:25.367273 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.367705 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.368645 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.368806 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.368852 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:25.370962 (+ 2110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.2:60329: BlockingRecv error: recv got EOF from 127.0.203.2:60329 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 19:08:25.371210 9808 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:08:25.440914 9811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:25.441112 9811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:25.441135 9811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:25.442240 9811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:25.442274 9811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:25.442301 9811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:08:25.443694 9811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:38687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/krb5kdc/oryx_127.0.203.3.keytab
--principal=oryx/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:42961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=33161
--tserver_master_addrs=127.0.203.62:43469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:25.443886 9811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:25.444058 9811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:25.449162 9816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:25.449222 9811 server_base.cc:1034] running on GCE node
W20260620 19:08:25.449190 9817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:25.449186 9819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:25.449522 9811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:25.449795 9811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:25.450996 9811 hybrid_clock.cc:648] HybridClock initialized: now 1781982505450973 us; error 34 us; skew 500 ppm
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982505, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:25.453562 9811 init.cc:377] Logged in from keytab as oryx/127.0.203.3@KRBTEST.COM (short username oryx)
I20260620 19:08:25.454068 9811 webserver.cc:460] Webserver started at http://127.0.203.3:33161/ using document root <none> and password file <none>
I20260620 19:08:25.454303 9811 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:25.454383 9811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:25.455814 9811 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:25.456423 9826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:25.456598 9811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:25.456700 9811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "135625f6d4dd41b8ae035cb488fefc52"
format_stamp: "Formatted at 2026-06-20 19:08:24 on dist-test-slave-wd30"
server_key: "76b636879277200adb5eff7d9bb36569"
server_key_iv: "8f69ced441d82ba648d10b0a78403553"
server_key_version: "encryptionkey@0"
I20260620 19:08:25.457074 9811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:25.464640 9811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:25.466436 9811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:25.466586 9811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:25.466825 9811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:25.467172 9811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:25.467231 9811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:25.467265 9811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:25.467295 9811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:25.473230 9811 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:42961
I20260620 19:08:25.473290 9939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:42961 every 8 connection(s)
I20260620 19:08:25.473603 9811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:25.480832 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 9811
W20260620 19:08:25.480893 9676 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:25.474281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:44661 (local address 127.0.203.62:43469)
0620 19:08:25.474324 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:25.474328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:25.474558 (+ 230us) server_negotiation.cc:407] Connection header received
0620 19:08:25.475083 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:25.475085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:25.475097 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:25.475181 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:25.476135 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.476660 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.477671 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.477870 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.477955 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:25.480822 (+ 2867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.3:44661: BlockingRecv error: recv got EOF from 127.0.203.3:44661 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:08:25.480877 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:25.474274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.3:44661)
0620 19:08:25.474435 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:25.474469 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:25.474975 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:25.475263 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:25.475267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:25.475841 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:25.476032 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:25.476036 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.476755 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.476757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:25.477578 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:25.477580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.477959 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.477960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:25.478089 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:25.478651 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:25.478659 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:25.480718 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
W20260620 19:08:25.481110 9940 heartbeater.cc:644] Failed to heartbeat to 127.0.203.62:43469 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.203.62:43469: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:25.486017 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:25.481300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.3:51833)
0620 19:08:25.481444 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:25.481453 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:25.481488 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:25.481636 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:25.481637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:25.481793 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:25.481860 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:25.481862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.482616 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.482617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:25.483346 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:25.483348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.483682 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.483683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:25.483754 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:25.484204 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:25.484209 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:25.485972 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128}
W20260620 19:08:25.486349 9678 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:25.481432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:51833 (local address 127.0.203.62:43469)
0620 19:08:25.481488 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:25.481490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:25.481497 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:08:25.481541 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:25.481542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:25.481552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:25.481614 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:25.481937 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.482540 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.483428 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.483617 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.483661 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:25.486276 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.3:51833: BlockingRecv error: recv got EOF from 127.0.203.3:51833 (error 108)
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":31}
I20260620 19:08:25.488612 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:25.481400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38870 (local address 127.0.203.62:43469)
0620 19:08:25.481477 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:25.481479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:25.481489 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:08:25.481542 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:25.481542 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:25.481552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:25.481611 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:25.482278 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.482813 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.483760 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.483964 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.484027 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:25.485230 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:25.485235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:25.485235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:25.485248 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:25.487313 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:25.487653 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:25.487655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:25.487655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:25.487742 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:25.487942 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:25.487943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:25.487943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:25.488138 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:25.488230 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:25.488423 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 19:08:25.488523 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:25 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:25.490718 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:25.486342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.3:51439)
0620 19:08:25.486382 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:25.486392 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:25.486429 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:25.486623 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:25.486623 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:25.486723 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:25.486745 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:25.486746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.487587 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.487588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:25.488253 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:25.488255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.488693 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.488694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:25.488764 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:25.489089 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:25.489093 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:25.490678 (+ 1585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 19:08:25.490834 9940 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:08:25.490847 9678 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:25.486393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:51439 (local address 127.0.203.62:43469)
0620 19:08:25.486423 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:25.486425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:25.486435 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:08:25.486533 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:25.486533 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:25.486545 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:25.486608 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:25.486815 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.487519 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.488343 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:25.488605 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:25.488675 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:25.490769 (+ 2094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.3:51439: BlockingRecv error: recv got EOF from 127.0.203.3:51439 (error 108)
Metrics: {"server-negotiator.queue_time_us":15,"thread_start_us":59,"threads_started":1}
Jun 20 19:08:26 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:26 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:26.252682 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:26.243778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.1:52645)
0620 19:08:26.243930 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:26.243941 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:26.243993 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:26.244149 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:26.244150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:26.244311 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:26.244366 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:26.244367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:26.245000 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:26.245000 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:26.245540 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:26.245541 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:26.245857 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:26.245857 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:26.245897 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:26.250696 (+ 4799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:26.250702 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:26.252614 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
W20260620 19:08:26.252851 9946 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:26.243849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:52645 (local address 127.0.203.62:43469)
0620 19:08:26.243979 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:26.243980 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:26.243991 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:08:26.244054 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:26.244055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:26.244065 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:26.244129 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:26.244425 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:26.244929 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:26.245609 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:26.245790 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:26.245840 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:26.252765 (+ 6925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.1:52645: BlockingRecv error: recv got EOF from 127.0.203.1:52645 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
Jun 20 19:08:26 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:26 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:26.376061 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:26.371596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.2:55951)
0620 19:08:26.371742 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:26.371760 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:26.371818 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:26.371929 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:26.371929 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:26.372084 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:26.372132 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:26.372133 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:26.372760 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:26.372761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:26.373300 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:26.373302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:26.373620 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:26.373621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:26.373661 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:26.374103 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:26.374107 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:26.375946 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
W20260620 19:08:26.376097 9946 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:26.371650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:55951 (local address 127.0.203.62:43469)
0620 19:08:26.371709 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:26.371711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:26.371812 (+ 101us) server_negotiation.cc:407] Connection header received
0620 19:08:26.371864 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:26.371865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:26.371875 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:26.371925 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:26.372183 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:26.372680 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:26.373369 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:26.373559 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:26.373609 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:26.376031 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.2:55951: BlockingRecv error: recv got EOF from 127.0.203.2:55951 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 19:08:26 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:26 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:26.496065 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:26.491248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.3:49829)
0620 19:08:26.491411 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:26.491423 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:26.491520 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:26.491672 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:26.491673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:26.491850 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:26.491895 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:26.491896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:26.492574 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:26.492576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:26.493119 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:26.493121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:26.493494 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:26.493496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:26.493555 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:26.494058 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:26.494063 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:26.495922 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
W20260620 19:08:26.496114 9946 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:26.491319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:49829 (local address 127.0.203.62:43469)
0620 19:08:26.491397 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:26.491399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:26.491501 (+ 102us) server_negotiation.cc:407] Connection header received
0620 19:08:26.491603 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:26.491604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:26.491616 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:26.491657 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:26.491985 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:26.492488 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:26.493207 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:26.493406 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:26.493484 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:26.496045 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.3:49829: BlockingRecv error: recv got EOF from 127.0.203.3:49829 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 19:08:27 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:27 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:27.257910 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:27.253250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.1:58469)
0620 19:08:27.253398 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:27.253409 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:27.253458 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:27.253635 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:27.253636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:27.253801 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:27.253848 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:27.253850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:27.254512 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:27.254513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:27.255030 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:27.255031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:27.255520 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:27.255521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:27.255562 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:27.256060 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:27.256064 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:27.257846 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
W20260620 19:08:27.258055 9950 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:27.253344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:58469 (local address 127.0.203.62:43469)
0620 19:08:27.253473 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:27.253475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:27.253485 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:08:27.253517 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:27.253518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:27.253530 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:27.253594 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:27.253930 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:27.254438 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:27.255100 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:27.255452 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:27.255508 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:27.257933 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.1:58469: BlockingRecv error: recv got EOF from 127.0.203.1:58469 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
Jun 20 19:08:27 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:27 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:27.381088 9951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:27.376595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.2:37915)
0620 19:08:27.376720 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:27.376732 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:27.376800 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:27.376937 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:27.376937 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:27.377126 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:27.377173 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:27.377175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:27.377866 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:27.377867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:27.378410 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:27.378412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:27.378787 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:27.378788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:27.378844 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:27.379303 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:27.379307 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:27.380984 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
W20260620 19:08:27.381122 9950 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:27.376673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:37915 (local address 127.0.203.62:43469)
0620 19:08:27.376721 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:27.376723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:27.376828 (+ 105us) server_negotiation.cc:407] Connection header received
0620 19:08:27.376855 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:27.376856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:27.376868 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:27.376922 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:27.377250 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:27.377772 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:27.378498 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:27.378701 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:27.378785 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:27.381059 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.2:37915: BlockingRecv error: recv got EOF from 127.0.203.2:37915 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 19:08:27 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:27 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:27.501374 9952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:27.496536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.3:32919)
0620 19:08:27.496677 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:27.496792 (+ 115us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:27.496848 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:27.497058 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:27.497059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:27.497233 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:27.497278 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:27.497280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:27.498012 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:27.498013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:27.498601 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:27.498602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:27.498957 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:27.498958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:27.499009 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:27.499541 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:27.499547 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:27.501271 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":59,"threads_started":1}
W20260620 19:08:27.501410 9950 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:27.496729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:32919 (local address 127.0.203.62:43469)
0620 19:08:27.496875 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:27.496878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:27.496897 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:08:27.496921 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:27.496923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:27.496936 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:27.497010 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:27.497363 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:27.497929 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:27.498682 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:27.498884 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:27.498961 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:27.501348 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.3:32919: BlockingRecv error: recv got EOF from 127.0.203.3:32919 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
Jun 20 19:08:28 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:28 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:28.263320 9953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:28.258383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.1:48357)
0620 19:08:28.258542 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:28.258554 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:28.258614 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:28.258771 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:28.258772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:28.258931 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:28.258981 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:28.258982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:28.259704 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:28.259705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:28.260295 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:28.260298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:28.260644 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:28.260645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:28.260694 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:28.261148 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:28.261153 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:28.263227 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
W20260620 19:08:28.263392 9954 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:28.258483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:48357 (local address 127.0.203.62:43469)
0620 19:08:28.258605 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:28.258607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:28.258616 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:08:28.258671 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:28.258671 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:28.258682 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:28.258754 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:28.259044 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:28.259626 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:28.260379 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:28.260576 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:28.260631 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:28.263312 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.1:48357: BlockingRecv error: recv got EOF from 127.0.203.1:48357 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
Jun 20 19:08:28 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:28 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:28.386377 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:28.381679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.2:48911)
0620 19:08:28.381864 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:28.381883 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:28.381979 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:28.382141 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:28.382142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:28.382336 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:28.382385 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:28.382387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:28.383064 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:28.383065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:28.383692 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:28.383694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:28.384068 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:28.384068 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:28.384120 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:28.384560 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:28.384564 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:28.386281 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":109,"threads_started":1}
W20260620 19:08:28.386426 9954 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:28.381757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:48911 (local address 127.0.203.62:43469)
0620 19:08:28.381812 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:28.381814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:28.381958 (+ 144us) server_negotiation.cc:407] Connection header received
0620 19:08:28.382046 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:28.382047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:28.382062 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:28.382133 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:28.382466 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:28.382974 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:28.383786 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:28.383982 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:28.384069 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:28.386362 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.2:48911: BlockingRecv error: recv got EOF from 127.0.203.2:48911 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 19:08:28 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:28 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:28.508442 9956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:28.501844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.3:58947)
0620 19:08:28.503033 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:28.503045 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:28.503160 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:28.503335 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:28.503336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:28.503548 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:28.503620 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:28.503622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:28.504741 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:28.504742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:28.505533 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:28.505536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:28.505951 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:28.505952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:28.506045 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:28.506592 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:28.506597 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:28.508382 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1170,"thread_start_us":69,"threads_started":1}
W20260620 19:08:28.508652 9954 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:28.502020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:58947 (local address 127.0.203.62:43469)
0620 19:08:28.502083 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:28.502085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:28.503143 (+ 1058us) server_negotiation.cc:407] Connection header received
0620 19:08:28.503227 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:28.503228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:28.503244 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:28.503315 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:28.503707 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:28.504624 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:28.505612 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:28.505864 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:28.505933 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:28.508557 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.3:58947: BlockingRecv error: recv got EOF from 127.0.203.3:58947 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 19:08:29 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:29 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:29.268896 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:29.263875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.1:50923)
0620 19:08:29.264065 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:29.264075 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:29.264163 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:29.264343 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:29.264344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:29.264524 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:29.264571 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:29.264573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:29.265314 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:29.265316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:29.265938 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:29.265940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:29.266307 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:29.266308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:29.266371 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:29.266909 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:29.266913 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:29.268796 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":112,"threads_started":1}
W20260620 19:08:29.268954 9957 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:29.263905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:50923 (local address 127.0.203.62:43469)
0620 19:08:29.264050 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:29.264052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:29.264144 (+ 92us) server_negotiation.cc:407] Connection header received
0620 19:08:29.264228 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:29.264229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:29.264249 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:29.264335 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:29.264675 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:29.265232 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:29.266020 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:29.266229 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:29.266306 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:29.268880 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.1:50923: BlockingRecv error: recv got EOF from 127.0.203.1:50923 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
Jun 20 19:08:29 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:29 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:29.391532 9959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:29.386952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.2:48017)
0620 19:08:29.387108 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:29.387158 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:29.387221 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:29.387414 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:29.387415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:29.387597 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:29.387646 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:29.387647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:29.388303 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:29.388304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:29.388841 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:29.388842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:29.389196 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:29.389197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:29.389235 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:29.389725 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:29.389730 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:29.391474 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
W20260620 19:08:29.391654 9957 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:29.387019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:48017 (local address 127.0.203.62:43469)
0620 19:08:29.387098 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:29.387100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:29.387211 (+ 111us) server_negotiation.cc:407] Connection header received
0620 19:08:29.387310 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:29.387311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:29.387323 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:29.387392 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:29.387733 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:29.388225 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:29.388932 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:29.389127 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:29.389188 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:29.391574 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.2:48017: BlockingRecv error: recv got EOF from 127.0.203.2:48017 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 19:08:29 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:29 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:29.513650 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:29.508875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.3:56759)
0620 19:08:29.509053 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:29.509066 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:29.509165 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:29.509345 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:29.509346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:29.509526 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:29.509572 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:29.509574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:29.510244 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:29.510245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:29.510783 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:29.510785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:29.511276 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:29.511277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:29.511333 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:29.511833 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:29.511838 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:29.513591 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
W20260620 19:08:29.513798 9957 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:29.508965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:56759 (local address 127.0.203.62:43469)
0620 19:08:29.509039 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:29.509041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:29.509145 (+ 104us) server_negotiation.cc:407] Connection header received
0620 19:08:29.509230 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:29.509231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:29.509253 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:29.509332 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:29.509655 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:29.510156 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:29.510894 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:29.511087 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:29.511188 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:29.513709 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.3:56759: BlockingRecv error: recv got EOF from 127.0.203.3:56759 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 19:08:30 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:30 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:30.273823 9961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:30.269377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.1:54595)
0620 19:08:30.269513 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:30.269531 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:30.269580 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:30.269818 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:30.269819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:30.269989 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:30.270038 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:30.270039 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:30.270695 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:30.270696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:30.271285 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:30.271287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:30.271613 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:30.271613 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:30.271652 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:30.272077 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:30.272082 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:30.273768 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":99,"threads_started":1}
W20260620 19:08:30.273978 9962 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:30.269553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:54595 (local address 127.0.203.62:43469)
0620 19:08:30.269692 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:30.269694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:30.269702 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:08:30.269731 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:30.269731 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:30.269741 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:30.269800 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:30.270099 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:30.270619 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:30.271352 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:30.271549 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:30.271602 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:30.273884 (+ 2282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.1:54595: BlockingRecv error: recv got EOF from 127.0.203.1:54595 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":94,"threads_started":1}
Jun 20 19:08:30 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:30 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:30.396633 9963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:30.392115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.2:58355)
0620 19:08:30.392265 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:30.392277 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:30.392356 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:30.392506 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:30.392507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:30.392681 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:30.392733 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:30.392735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:30.393396 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:30.393410 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:30.393941 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:30.393942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:30.394276 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:30.394276 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:30.394316 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:30.394784 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:30.394789 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:30.396573 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
W20260620 19:08:30.396747 9962 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:30.392252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:58355 (local address 127.0.203.62:43469)
0620 19:08:30.392359 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:30.392361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:30.392402 (+ 41us) server_negotiation.cc:407] Connection header received
0620 19:08:30.392428 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:30.392429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:30.392440 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:30.392490 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:30.392807 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:30.393298 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:30.394025 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:30.394208 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:30.394261 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:30.396671 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.2:58355: BlockingRecv error: recv got EOF from 127.0.203.2:58355 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 19:08:30 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:30 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:30.518628 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:30.514261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.3:53335)
0620 19:08:30.514385 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:30.514397 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:30.514469 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:30.514603 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:30.514603 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:30.514761 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:30.514810 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:30.514811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:30.515451 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:30.515451 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:30.515973 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:30.515974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:30.516289 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:30.516290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:30.516329 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:30.516831 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:30.516836 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:30.518559 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
W20260620 19:08:30.518764 9962 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:30.514348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:53335 (local address 127.0.203.62:43469)
0620 19:08:30.514445 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:30.514448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:30.514460 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:08:30.514519 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:30.514520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:30.514533 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:30.514586 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:30.514871 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:30.515381 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:30.516053 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:30.516226 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:30.516277 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:30.518657 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.3:53335: BlockingRecv error: recv got EOF from 127.0.203.3:53335 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 19:08:31 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:31 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:31.282408 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:31.274347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.1:47809)
0620 19:08:31.274485 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:31.274495 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:31.274538 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:31.274757 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:31.274758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:31.274917 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:31.274965 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:31.274966 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.275670 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.275671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:31.276189 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:31.276191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.276543 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.276544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:31.276603 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:31.280454 (+ 3851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:31.280460 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:31.282336 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":58,"threads_started":1}
W20260620 19:08:31.282552 9966 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:31.274479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:47809 (local address 127.0.203.62:43469)
0620 19:08:31.274621 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:31.274622 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:31.274630 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:08:31.274655 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:31.274655 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:31.274666 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:31.274741 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:31.275031 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.275601 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.276269 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.276471 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.276530 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:31.282454 (+ 5924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.1:47809: BlockingRecv error: recv got EOF from 127.0.203.1:47809 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
Jun 20 19:08:31 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:31 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:31.401984 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:31.397246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.2:50099)
0620 19:08:31.397386 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:31.397404 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:31.397464 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:31.397615 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:31.397616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:31.397774 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:31.397822 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:31.397824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.398764 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.398764 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:31.399368 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:31.399371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.399693 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.399694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:31.399747 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:31.400175 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:31.400180 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:31.401916 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
W20260620 19:08:31.402124 9966 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:31.397323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:50099 (local address 127.0.203.62:43469)
0620 19:08:31.397406 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:31.397408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:31.397466 (+ 58us) server_negotiation.cc:407] Connection header received
0620 19:08:31.397511 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:31.397512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:31.397528 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:31.397590 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:31.397883 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.398697 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.399436 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.399627 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.399684 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:31.402012 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.2:50099: BlockingRecv error: recv got EOF from 127.0.203.2:50099 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 19:08:31 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:31 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:31.523732 9968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:31.519156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.3:33599)
0620 19:08:31.519307 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:31.519325 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:31.519380 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:31.519514 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:31.519514 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:31.519673 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:31.519720 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:31.519721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.520371 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.520372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:31.520884 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:31.520886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.521294 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.521296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:31.521346 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:31.521820 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:31.521825 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:31.523673 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
W20260620 19:08:31.523852 9966 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:31.519266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:33599 (local address 127.0.203.62:43469)
0620 19:08:31.519313 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:31.519315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:31.519385 (+ 70us) server_negotiation.cc:407] Connection header received
0620 19:08:31.519435 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:31.519435 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:31.519447 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:31.519498 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:31.519791 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.520303 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.520975 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:31.521200 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:31.521261 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:31.523774 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.3:33599: BlockingRecv error: recv got EOF from 127.0.203.3:33599 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 19:08:32 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:32 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:32.287528 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:32.282903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.1:51413)
0620 19:08:32.283060 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:32.283073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:32.283213 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:32.283422 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:32.283423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:32.283585 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:32.283640 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:32.283641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.284319 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.284320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:32.284849 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:32.284851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.285186 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.285187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:32.285229 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:32.285680 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:32.285684 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:32.287468 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
W20260620 19:08:32.287642 9970 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:32.282979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:51413 (local address 127.0.203.62:43469)
0620 19:08:32.283090 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:32.283091 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:32.283184 (+ 93us) server_negotiation.cc:407] Connection header received
0620 19:08:32.283297 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:32.283298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:32.283310 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:32.283370 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:32.283727 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.284223 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.284925 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.285108 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.285157 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:32.287566 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.1:51413: BlockingRecv error: recv got EOF from 127.0.203.1:51413 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":49,"threads_started":1}
Jun 20 19:08:32 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:32 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:32.407367 9971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:32.402600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.2:45749)
0620 19:08:32.402755 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:32.402767 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:32.402830 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:32.402946 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:32.402947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:32.403190 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:32.403246 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:32.403248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.403870 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.403871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:32.404475 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:32.404478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.404814 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.404815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:32.404855 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:32.405542 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:32.405548 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:32.407311 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
W20260620 19:08:32.407487 9970 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:32.402623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:45749 (local address 127.0.203.62:43469)
0620 19:08:32.402699 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:32.402700 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:32.402830 (+ 130us) server_negotiation.cc:407] Connection header received
0620 19:08:32.402872 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:32.402873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:32.402883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:32.402943 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:32.403308 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.403804 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.404543 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.404749 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.404805 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:32.407409 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.2:45749: BlockingRecv error: recv got EOF from 127.0.203.2:45749 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 19:08:32 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:32 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:32.528554 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:32.524206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.3:38567)
0620 19:08:32.524349 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:32.524361 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:32.524418 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:32.524545 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:32.524546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:32.524706 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:32.524755 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:32.524757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.525386 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.525386 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:32.525899 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:32.525901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.526234 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.526235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:32.526297 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:32.526740 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:32.526745 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:32.528494 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
W20260620 19:08:32.528656 9970 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:32.524275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:38567 (local address 127.0.203.62:43469)
0620 19:08:32.524328 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:32.524330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:32.524405 (+ 75us) server_negotiation.cc:407] Connection header received
0620 19:08:32.524471 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:32.524472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:32.524484 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:32.524524 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:32.524819 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.525315 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.525970 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:32.526146 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:32.526195 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:32.528587 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.3:38567: BlockingRecv error: recv got EOF from 127.0.203.3:38567 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 19:08:33 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:33 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:33.292433 9973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:33.288123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.1:50671)
0620 19:08:33.288265 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:33.288277 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:33.288331 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:33.288511 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:33.288512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:33.288669 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:33.288715 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:33.288717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:33.289349 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:33.289349 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:33.289874 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:33.289875 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:33.290194 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:33.290194 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:33.290232 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:33.290669 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:33.290673 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:33.292376 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
W20260620 19:08:33.292558 9974 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:33.288244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:50671 (local address 127.0.203.62:43469)
0620 19:08:33.288373 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:33.288375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:33.288392 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:08:33.288413 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:33.288414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:33.288424 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:33.288486 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:33.288787 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:33.289283 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:33.289946 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:33.290131 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:33.290181 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:33.292474 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.1:50671: BlockingRecv error: recv got EOF from 127.0.203.1:50671 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
Jun 20 19:08:33 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:33 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:33.412752 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:33.407818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.2:51259)
0620 19:08:33.408002 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:33.408014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:33.408102 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:33.408254 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:33.408255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:33.408459 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:33.408505 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:33.408507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:33.409322 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:33.409323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:33.409858 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:33.409860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:33.410257 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:33.410258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:33.410330 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:33.410830 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:33.410835 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:33.412654 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":100,"threads_started":1}
W20260620 19:08:33.412810 9974 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:33.407903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:51259 (local address 127.0.203.62:43469)
0620 19:08:33.407981 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:33.407983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:33.408084 (+ 101us) server_negotiation.cc:407] Connection header received
0620 19:08:33.408176 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:33.408177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:33.408187 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:33.408247 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:33.408591 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:33.409235 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:33.409950 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:33.410152 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:33.410255 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:33.412735 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.2:51259: BlockingRecv error: recv got EOF from 127.0.203.2:51259 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 19:08:33 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:33 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:33.534181 9976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:33.529101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.3:44747)
0620 19:08:33.529409 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:33.529422 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:33.529515 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:33.529691 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:33.529692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:33.529867 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:33.529914 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:33.529916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:33.530616 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:33.530618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:33.531240 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:33.531243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:33.531605 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:33.531606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:33.531658 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:33.532148 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:33.532156 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:33.534124 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":98,"threads_started":1}
W20260620 19:08:33.534267 9974 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:33.529221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:44747 (local address 127.0.203.62:43469)
0620 19:08:33.529409 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:33.529411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:33.529495 (+ 84us) server_negotiation.cc:407] Connection header received
0620 19:08:33.529590 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:33.529592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:33.529607 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:33.529685 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:33.530006 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:33.530517 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:33.531323 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:33.531519 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:33.531603 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:33.534220 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.3:44747: BlockingRecv error: recv got EOF from 127.0.203.3:44747 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
Jun 20 19:08:34 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:34 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:34.298066 9977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:34.293022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.1:37241)
0620 19:08:34.293268 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:34.293284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:34.293343 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:34.293501 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:34.293502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:34.293668 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:34.293718 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:34.293719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:34.294407 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:34.294407 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:34.294940 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:34.294941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:34.295449 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:34.295450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:34.295538 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:34.296068 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:34.296073 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:34.298007 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":188,"threads_started":1}
W20260620 19:08:34.298215 9978 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:34.293121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:37241 (local address 127.0.203.62:43469)
0620 19:08:34.293268 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:34.293270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:34.293330 (+ 60us) server_negotiation.cc:407] Connection header received
0620 19:08:34.293391 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:34.293391 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:34.293402 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:34.293472 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:34.293796 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:34.294323 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:34.295044 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:34.295355 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:34.295431 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:34.298126 (+ 2695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.1:37241: BlockingRecv error: recv got EOF from 127.0.203.1:37241 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
Jun 20 19:08:34 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:34 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:34.418200 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:34.413351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.2:58791)
0620 19:08:34.413579 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:34.413591 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:34.413651 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:34.413809 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:34.413810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:34.413976 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:34.414025 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:34.414026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:34.414663 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:34.414664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:34.415262 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:34.415264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:34.415590 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:34.415590 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:34.415627 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:34.416055 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:34.416059 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:34.418141 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":119,"threads_started":1}
W20260620 19:08:34.418380 9978 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:34.413508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:58791 (local address 127.0.203.62:43469)
0620 19:08:34.413624 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:34.413626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:34.413638 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:08:34.413717 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:34.413718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:34.413730 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:34.413788 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:34.414091 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:34.414592 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:34.415334 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:34.415526 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:34.415577 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:34.418277 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.2:58791: BlockingRecv error: recv got EOF from 127.0.203.2:58791 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 20 19:08:34 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:34 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:34.539927 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:34.534750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.3:39271)
0620 19:08:34.534963 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:34.534978 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:34.535088 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:34.535321 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:34.535322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:34.535513 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:34.535561 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:34.535563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:34.536265 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:34.536267 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:34.536799 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:34.536801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:34.537190 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:34.537192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:34.537256 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:34.537785 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:34.537791 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:34.539829 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":108,"threads_started":1}
W20260620 19:08:34.539986 9978 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:34.534886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:39271 (local address 127.0.203.62:43469)
0620 19:08:34.534972 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:34.534974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:34.535057 (+ 83us) server_negotiation.cc:407] Connection header received
0620 19:08:34.535221 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:34.535222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:34.535238 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:34.535314 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:34.535646 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:34.536179 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:34.536885 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:34.537091 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:34.537187 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:34.539922 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.3:39271: BlockingRecv error: recv got EOF from 127.0.203.3:39271 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 19:08:35 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:35 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:08:35.305939 9982 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:35.298647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:36309 (local address 127.0.203.62:43469)
0620 19:08:35.298787 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:35.298789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:35.298803 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:08:35.298840 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:35.298840 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:35.298855 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:35.298951 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:35.299516 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:35.300384 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:35.301917 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:35.302294 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:35.302363 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:35.305880 (+ 3517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.1:36309: BlockingRecv error: recv got EOF from 127.0.203.1:36309 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
I20260620 19:08:35.305929 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:35.298550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.1:36309)
0620 19:08:35.298692 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:35.298704 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:35.298788 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:35.299022 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:35.299024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:35.299299 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:35.299375 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:35.299378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:35.300802 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:35.300803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:35.301821 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:35.301824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:35.302446 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:35.302447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:35.302526 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:35.303270 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:35.303276 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:35.305788 (+ 2512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
Jun 20 19:08:35 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:35 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:35.423597 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:35.418652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.2:32907)
0620 19:08:35.418837 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:35.418851 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:35.418911 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:35.419047 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:35.419048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:35.419263 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:35.419315 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:35.419317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:35.419968 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:35.419969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:35.420508 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:35.420510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:35.420839 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:35.420840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:35.420885 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:35.421321 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:35.421325 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:35.423484 (+ 2159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":105,"threads_started":1}
W20260620 19:08:35.423629 9982 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:35.418747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:32907 (local address 127.0.203.62:43469)
0620 19:08:35.418820 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:35.418822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:35.418913 (+ 91us) server_negotiation.cc:407] Connection header received
0620 19:08:35.418959 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:35.418960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:35.418973 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:35.419028 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:35.419375 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:35.419886 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:35.420581 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:35.420770 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:35.420825 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:35.423569 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.2:32907: BlockingRecv error: recv got EOF from 127.0.203.2:32907 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 19:08:35 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:35 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:35.545058 9984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:35.540423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.3:56699)
0620 19:08:35.540566 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:35.540577 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:35.540631 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:35.540825 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:35.540826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:35.540997 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:35.541044 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:35.541046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:35.541744 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:35.541744 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:35.542256 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:35.542257 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:35.542648 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:35.542648 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:35.542688 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:35.543166 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:35.543171 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:35.544998 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
W20260620 19:08:35.545207 9982 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:35.540606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:56699 (local address 127.0.203.62:43469)
0620 19:08:35.540700 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:35.540701 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:35.540713 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:08:35.540737 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:35.540737 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:35.540748 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:35.540806 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:35.541137 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:35.541669 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:35.542371 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:35.542572 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:35.542636 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:35.545127 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.3:56699: BlockingRecv error: recv got EOF from 127.0.203.3:56699 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 19:08:36 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:36 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:36.314872 9985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:36.306404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.1:57045)
0620 19:08:36.306542 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:36.306554 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:36.306610 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:36.306767 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:36.306768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:36.306940 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:36.306990 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:36.306991 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:36.307655 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:36.307656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:36.308226 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:36.308228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:36.308667 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:36.308668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:36.308753 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:36.312860 (+ 4107us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:36.312866 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:36.314811 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
W20260620 19:08:36.315091 9986 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:36.306493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:57045 (local address 127.0.203.62:43469)
0620 19:08:36.306626 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:36.306628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:36.306662 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:08:36.306682 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:36.306683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:36.306694 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:36.306747 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:36.307056 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:36.307589 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:36.308356 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:36.308560 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:36.308636 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:36.314998 (+ 6362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.1:57045: BlockingRecv error: recv got EOF from 127.0.203.1:57045 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
Jun 20 19:08:36 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:36 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:36.429437 9987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:36.424142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.2:34339)
0620 19:08:36.424372 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:36.424384 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:36.424489 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:36.424660 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:36.424661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:36.424860 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:36.424907 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:36.424909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:36.425604 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:36.425605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:36.426294 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:36.426299 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:36.426882 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:36.426883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:36.426931 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:36.427439 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:36.427444 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:36.429362 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":117,"threads_started":1}
W20260620 19:08:36.429560 9986 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:36.424243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:34339 (local address 127.0.203.62:43469)
0620 19:08:36.424349 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:36.424352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:36.424469 (+ 117us) server_negotiation.cc:407] Connection header received
0620 19:08:36.424559 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:36.424560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:36.424575 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:36.424654 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:36.425006 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:36.425514 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:36.426429 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:36.426805 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:36.426868 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:36.429459 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.2:34339: BlockingRecv error: recv got EOF from 127.0.203.2:34339 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 19:08:36 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:36 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:36.550573 9988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:36.545626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.3:37709)
0620 19:08:36.545753 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:36.545767 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:36.545843 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:36.546037 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:36.546038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:36.546297 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:36.546352 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:36.546354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:36.547052 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:36.547053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:36.547597 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:36.547599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:36.547941 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:36.547941 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:36.547981 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:36.548445 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:36.548450 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:36.550513 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
W20260620 19:08:36.550719 9986 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:36.545695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:37709 (local address 127.0.203.62:43469)
0620 19:08:36.545779 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:36.545781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:36.545830 (+ 49us) server_negotiation.cc:407] Connection header received
0620 19:08:36.545934 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:36.545935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:36.545952 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:36.546014 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:36.546450 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:36.546973 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:36.547685 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:36.547871 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:36.547930 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:36.550630 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.3:37709: BlockingRecv error: recv got EOF from 127.0.203.3:37709 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 19:08:37 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:37 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:37.320345 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:37.315455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.1:43361)
0620 19:08:37.315620 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:37.315632 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:37.315691 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:37.315883 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:37.315884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:37.316078 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:37.316133 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:37.316135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:37.316801 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:37.316802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:37.317338 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:37.317339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:37.317679 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:37.317680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:37.317723 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:37.318179 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:37.318184 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:37.320260 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":58,"threads_started":1}
W20260620 19:08:37.320437 9990 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:37.315609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:43361 (local address 127.0.203.62:43469)
0620 19:08:37.315727 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:37.315729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:37.315737 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:08:37.315758 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:37.315759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:37.315769 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:37.315834 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:37.316220 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:37.316720 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:37.317418 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:37.317604 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:37.317660 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:37.320366 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.1:43361: BlockingRecv error: recv got EOF from 127.0.203.1:43361 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
Jun 20 19:08:37 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:37 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:37.434499 9991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:37.429930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.2:39821)
0620 19:08:37.430076 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:37.430089 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:37.430166 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:37.430291 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:37.430292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:37.430466 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:37.430518 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:37.430520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:37.431206 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:37.431207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:37.431740 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:37.431742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:37.432179 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:37.432180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:37.432224 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:37.432677 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:37.432682 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:37.434430 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":57,"threads_started":1}
W20260620 19:08:37.434625 9990 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:37.430051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:39821 (local address 127.0.203.62:43469)
0620 19:08:37.430120 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:37.430122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:37.430149 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:08:37.430211 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:37.430212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:37.430223 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:37.430270 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:37.430583 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:37.431078 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:37.431825 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:37.432095 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:37.432164 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:37.434525 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.2:39821: BlockingRecv error: recv got EOF from 127.0.203.2:39821 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 19:08:37 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:37 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:37.555691 9992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:37.551192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.3:40477)
0620 19:08:37.551315 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:37.551331 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:37.551389 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:37.551542 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:37.551543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:37.551713 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:37.551765 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:37.551767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:37.552404 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:37.552405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:37.552961 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:37.552963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:37.553277 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:37.553278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:37.553317 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:37.553777 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:37.553781 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:37.555622 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
W20260620 19:08:37.555835 9990 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:37.551229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:40477 (local address 127.0.203.62:43469)
0620 19:08:37.551293 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:37.551295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:37.551389 (+ 94us) server_negotiation.cc:407] Connection header received
0620 19:08:37.551462 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:37.551463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:37.551476 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:37.551517 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:37.551828 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:37.552325 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:37.553036 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:37.553212 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:37.553262 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:37.555733 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.3:40477: BlockingRecv error: recv got EOF from 127.0.203.3:40477 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 19:08:38 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:38 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:38.325578 9993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:38.320979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.1:34199)
0620 19:08:38.321108 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:38.321125 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:38.321176 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:38.321389 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:38.321390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:38.321558 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:38.321606 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:38.321607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:38.322255 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:38.322256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:38.322781 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:38.322782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:38.323114 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:38.323115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:38.323223 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:38.323731 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:38.323736 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:38.325517 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
W20260620 19:08:38.325731 9994 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:38.321081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:34199 (local address 127.0.203.62:43469)
0620 19:08:38.321239 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:38.321243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:38.321257 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:08:38.321281 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:38.321282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:38.321294 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:38.321380 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:38.321678 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:38.322180 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:38.322856 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:38.323048 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:38.323100 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:38.325636 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.1:34199: BlockingRecv error: recv got EOF from 127.0.203.1:34199 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
Jun 20 19:08:38 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:38 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:38.439580 9995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:38.434898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.2:43139)
0620 19:08:38.435043 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:38.435076 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:38.435193 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:38.435349 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:38.435350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:38.435537 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:38.435583 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:38.435585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:38.436285 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:38.436286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:38.436819 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:38.436821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:38.437224 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:38.437225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:38.437292 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:38.437708 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:38.437712 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:38.439482 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
W20260620 19:08:38.439628 9994 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:38.435024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:43139 (local address 127.0.203.62:43469)
0620 19:08:38.435098 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:38.435100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:38.435159 (+ 59us) server_negotiation.cc:407] Connection header received
0620 19:08:38.435275 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:38.435276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:38.435289 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:38.435344 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:38.435666 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:38.436193 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:38.436927 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:38.437139 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:38.437224 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:38.439564 (+ 2340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.2:43139: BlockingRecv error: recv got EOF from 127.0.203.2:43139 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 19:08:38 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:38 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:38.560659 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:38.556194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.3:52135)
0620 19:08:38.556347 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:38.556359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:38.556417 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:38.556527 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:38.556527 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:38.556692 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:38.556736 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:38.556737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:38.557357 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:38.557357 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:38.557866 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:38.557867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:38.558192 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:38.558192 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:38.558231 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:38.558673 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:38.558678 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:38.560590 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
W20260620 19:08:38.560947 9994 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:38.556246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:52135 (local address 127.0.203.62:43469)
0620 19:08:38.556313 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:38.556315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:38.556419 (+ 104us) server_negotiation.cc:407] Connection header received
0620 19:08:38.556467 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:38.556468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:38.556479 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:38.556515 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:38.556795 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:38.557285 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:38.557944 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:38.558125 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:38.558179 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:38.560890 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.3:52135: BlockingRecv error: recv got EOF from 127.0.203.3:52135 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 19:08:39 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:39 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:39.330818 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:39.326130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.1:49551)
0620 19:08:39.326318 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:39.326336 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:39.326405 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:39.326564 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:39.326565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:39.326733 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:39.326782 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:39.326784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:39.327514 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:39.327515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:39.328074 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:39.328076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:39.328438 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:39.328439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:39.328492 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:39.328964 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:39.328968 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:39.330762 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":80,"threads_started":1}
W20260620 19:08:39.330919 9998 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:39.326201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:49551 (local address 127.0.203.62:43469)
0620 19:08:39.326374 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:39.326376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:39.326388 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:08:39.326467 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:39.326468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:39.326478 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:39.326539 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:39.326851 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:39.327439 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:39.328156 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:39.328350 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:39.328406 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:39.330843 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.1:49551: BlockingRecv error: recv got EOF from 127.0.203.1:49551 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":107,"threads_started":1}
Jun 20 19:08:39 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:39 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:39.444308 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:39.440066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.2:41937)
0620 19:08:39.440201 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:39.440212 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:39.440268 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:39.440382 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:39.440383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:39.440553 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:39.440598 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:39.440600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:39.441212 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:39.441213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:39.441735 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:39.441736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:39.442046 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:39.442047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:39.442086 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:39.442599 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:39.442604 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:39.444236 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
W20260620 19:08:39.444378 9998 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:39.440138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:41937 (local address 127.0.203.62:43469)
0620 19:08:39.440224 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:39.440226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:39.440254 (+ 28us) server_negotiation.cc:407] Connection header received
0620 19:08:39.440323 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:39.440324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:39.440334 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:39.440368 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:39.440661 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:39.441146 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:39.441806 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:39.441986 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:39.442035 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:39.444332 (+ 2297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.2:41937: BlockingRecv error: recv got EOF from 127.0.203.2:41937 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 19:08:39 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:39 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.3@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:39.565918 10000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:39.561280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.3:54253)
0620 19:08:39.561439 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:39.561450 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:39.561507 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:39.561673 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:39.561673 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:39.561886 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:39.561946 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:39.561948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:39.562629 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:39.562630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:39.563200 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:39.563202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:39.563528 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:39.563529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:39.563566 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:39.564097 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:39.564102 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:39.565850 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
W20260620 19:08:39.565964 9998 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:39.561341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:54253 (local address 127.0.203.62:43469)
0620 19:08:39.561406 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:39.561408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:39.561557 (+ 149us) server_negotiation.cc:407] Connection header received
0620 19:08:39.561584 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:39.561585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:39.561599 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:39.561656 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:39.562050 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:39.562537 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:39.563277 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:39.563460 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:39.563515 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:39.565918 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.3:54253: BlockingRecv error: recv got EOF from 127.0.203.3:54253 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 19:08:40 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:40 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.1@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:40.336021 10001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:40.331424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.1:60651)
0620 19:08:40.331564 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:40.331576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:40.331639 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:40.331805 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:40.331805 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:40.331964 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:40.332013 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:40.332014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:40.332735 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:40.332735 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:40.333264 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:40.333266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:40.333637 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:40.333638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:40.333679 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:40.334102 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:40.334106 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:40.335964 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
W20260620 19:08:40.336177 10002 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:40.331534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:60651 (local address 127.0.203.62:43469)
0620 19:08:40.331666 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:40.331668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:40.331676 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:08:40.331699 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:40.331699 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:40.331710 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:40.331785 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:40.332075 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:40.332666 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:40.333339 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:40.333560 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:40.333621 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:40.336077 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.1:60651: BlockingRecv error: recv got EOF from 127.0.203.1:60651 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
Jun 20 19:08:40 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:08:40 dist-test-slave-wd30 krb5kdc[8942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.2@KRBTEST.COM for oryx/127.0.203.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:08:40.450163 10003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:40.444977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:43469 (local address 127.0.203.2:34501)
0620 19:08:40.445260 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:40.445271 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:40.445338 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:40.445502 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:40.445503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:40.445666 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:40.445716 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:40.445717 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:40.446384 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:40.446385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:40.446926 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:40.446928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:40.447397 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:40.447398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:40.447448 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:40.447904 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:40.447908 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:40.450087 (+ 2179us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.62:43469: Server oryx/127.0.203.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":210,"threads_started":1}
W20260620 19:08:40.450276 10002 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:40.445158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:34501 (local address 127.0.203.62:43469)
0620 19:08:40.445234 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:40.445236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:40.445374 (+ 138us) server_negotiation.cc:407] Connection header received
0620 19:08:40.445410 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:40.445411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:40.445423 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:40.445479 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:40.445783 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:40.446277 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:40.447039 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:40.447317 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:40.447382 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:40.450203 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.2:34501: BlockingRecv error: recv got EOF from 127.0.203.2:34501 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:08:40.481623 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 9525
I20260620 19:08:40.486243 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 9677
I20260620 19:08:40.490520 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 9811
I20260620 19:08:40.494645 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 9455
2026-06-20T19:08:40Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17437 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T19:08:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:08:40Z Disabled control of system clock
I20260620 19:08:40.524888 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:44551
--webserver_interface=127.0.203.62
--webserver_port=0
--builtin_ntp_servers=127.0.203.20:39433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:44551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 19:08:40.605773 10007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:40.606019 10007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:40.606053 10007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:40.607270 10007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:40.607342 10007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:40.607364 10007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:40.607388 10007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:40.607414 10007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:08:40.608747 10007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:39433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:44551
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:44551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:40.609016 10007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:40.609338 10007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:40.614581 10013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:40.614610 10015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:40.614658 10012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:40.614688 10007 server_base.cc:1034] running on GCE node
I20260620 19:08:40.615039 10007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:40.615424 10007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:40.616586 10007 hybrid_clock.cc:648] HybridClock initialized: now 1781982520616565 us; error 36 us; skew 500 ppm
I20260620 19:08:40.617898 10007 webserver.cc:460] Webserver started at http://127.0.203.62:41903/ using document root <none> and password file <none>
I20260620 19:08:40.618188 10007 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:40.618240 10007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:40.618428 10007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:40.619625 10007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "57debb6a975b469190ecbe77cbbaf35d"
format_stamp: "Formatted at 2026-06-20 19:08:40 on dist-test-slave-wd30"
server_key: "b6d8ae7c7707f66b0b58a059a936e2f6"
server_key_iv: "88feea755b1a6fefa4b3331387c75fdf"
server_key_version: "encryptionkey@0"
I20260620 19:08:40.620014 10007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "57debb6a975b469190ecbe77cbbaf35d"
format_stamp: "Formatted at 2026-06-20 19:08:40 on dist-test-slave-wd30"
server_key: "b6d8ae7c7707f66b0b58a059a936e2f6"
server_key_iv: "88feea755b1a6fefa4b3331387c75fdf"
server_key_version: "encryptionkey@0"
I20260620 19:08:40.621413 10007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:08:40.622177 10021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:40.622362 10007 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:40.622448 10007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "57debb6a975b469190ecbe77cbbaf35d"
format_stamp: "Formatted at 2026-06-20 19:08:40 on dist-test-slave-wd30"
server_key: "b6d8ae7c7707f66b0b58a059a936e2f6"
server_key_iv: "88feea755b1a6fefa4b3331387c75fdf"
server_key_version: "encryptionkey@0"
I20260620 19:08:40.622503 10007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:40.656507 10007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:40.656859 10007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:40.657023 10007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:40.661073 10007 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:44551
I20260620 19:08:40.661134 10073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:44551 every 8 connection(s)
I20260620 19:08:40.661515 10007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:40.662312 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:40.665807 10074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d: Bootstrap starting.
I20260620 19:08:40.666365 10074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:40.666610 10074 log.cc:824] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:40.667338 10074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d: No bootstrap required, opened a new log
I20260620 19:08:40.668881 10074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57debb6a975b469190ecbe77cbbaf35d" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44551 } }
I20260620 19:08:40.669027 10074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:40.669050 10074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57debb6a975b469190ecbe77cbbaf35d, State: Initialized, Role: FOLLOWER
I20260620 19:08:40.669209 10074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57debb6a975b469190ecbe77cbbaf35d" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44551 } }
I20260620 19:08:40.669287 10074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:40.669312 10074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:40.669338 10074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:40.669528 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 10007
I20260620 19:08:40.669641 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:08:40.669834 812 external_mini_cluster.cc:1428] Setting key 9cf284565d2ddc4121728a73831cc8dc
I20260620 19:08:40.670047 10074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57debb6a975b469190ecbe77cbbaf35d" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44551 } }
I20260620 19:08:40.670166 10074 leader_election.cc:302] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57debb6a975b469190ecbe77cbbaf35d; no voters:
I20260620 19:08:40.670389 10074 leader_election.cc:288] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:08:40.670461 10078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:40.670594 10078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d [term 1 LEADER]: Becoming Leader. State: Replica: 57debb6a975b469190ecbe77cbbaf35d, State: Running, Role: LEADER
I20260620 19:08:40.670697 10074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:40.670679 10078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57debb6a975b469190ecbe77cbbaf35d" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44551 } }
I20260620 19:08:40.671479 10080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57debb6a975b469190ecbe77cbbaf35d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57debb6a975b469190ecbe77cbbaf35d" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44551 } } }
I20260620 19:08:40.671568 10080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:40.671728 10081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57debb6a975b469190ecbe77cbbaf35d. Latest consensus state: current_term: 1 leader_uuid: "57debb6a975b469190ecbe77cbbaf35d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57debb6a975b469190ecbe77cbbaf35d" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44551 } } }
I20260620 19:08:40.671777 10081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d [sys.catalog]: This master's current role is: LEADER
W20260620 19:08:40.672763 10095 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:08:40.672816 10095 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:08:40.672865 10088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:08:40.673069 10088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:08:40.674505 10088 catalog_manager.cc:1347] Generated new cluster ID: beffd813fa2344f0a32396d4cb5ac035
I20260620 19:08:40.674558 10088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:08:40.676343 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:40.670391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60942 (local address 127.0.203.62:44551)
0620 19:08:40.670751 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:40.670762 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:40.670794 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:08:40.670915 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:40.670921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:40.671006 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:40.671155 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:40.671497 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:40.672996 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:40.674108 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:40.674709 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:40.674838 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:40.675592 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:40.675596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:40.675596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:40.675814 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:40.675858 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:40.675873 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:40.675948 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:40.676022 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 19:08:40.676102 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":109,"threads_started":1}
I20260620 19:08:40.687901 10088 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:08:40.688614 10088 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:40.699906 10088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57debb6a975b469190ecbe77cbbaf35d: Generated new TSK 0
I20260620 19:08:40.700133 10088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:08:40.703996 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44551
--builtin_ntp_servers=127.0.203.20:39433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 19:08:40.786152 10098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:40.786397 10098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:40.786432 10098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:40.787663 10098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:40.787732 10098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:40.787775 10098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:08:40.789145 10098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:39433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:40.789399 10098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:40.789706 10098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:40.795661 10104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:40.795709 10103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:40.795661 10106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:40.796108 10098 server_base.cc:1034] running on GCE node
I20260620 19:08:40.796306 10098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:40.796553 10098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:40.797734 10098 hybrid_clock.cc:648] HybridClock initialized: now 1781982520797721 us; error 30 us; skew 500 ppm
I20260620 19:08:40.798982 10098 webserver.cc:460] Webserver started at http://127.0.203.1:33833/ using document root <none> and password file <none>
I20260620 19:08:40.799293 10098 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:40.799368 10098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:40.799527 10098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:40.800606 10098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "4a89ee16b97a49998f8526d41e103cb1"
format_stamp: "Formatted at 2026-06-20 19:08:40 on dist-test-slave-wd30"
server_key: "9cb835e0c65fb689e002ac89c04f2f0e"
server_key_iv: "69059448e39377b5846562d61bd59f23"
server_key_version: "encryptionkey@0"
I20260620 19:08:40.800985 10098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "4a89ee16b97a49998f8526d41e103cb1"
format_stamp: "Formatted at 2026-06-20 19:08:40 on dist-test-slave-wd30"
server_key: "9cb835e0c65fb689e002ac89c04f2f0e"
server_key_iv: "69059448e39377b5846562d61bd59f23"
server_key_version: "encryptionkey@0"
I20260620 19:08:40.802328 10098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:08:40.803136 10112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:40.803328 10098 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:40.803427 10098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "4a89ee16b97a49998f8526d41e103cb1"
format_stamp: "Formatted at 2026-06-20 19:08:40 on dist-test-slave-wd30"
server_key: "9cb835e0c65fb689e002ac89c04f2f0e"
server_key_iv: "69059448e39377b5846562d61bd59f23"
server_key_version: "encryptionkey@0"
I20260620 19:08:40.803515 10098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:40.812981 10098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:40.813285 10098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:40.813432 10098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:40.813659 10098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:40.813994 10098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:40.814049 10098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:40.814088 10098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:40.814117 10098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:40.819988 10098 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:40385
I20260620 19:08:40.820057 10225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:40385 every 8 connection(s)
I20260620 19:08:40.820398 10098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:08:40.825834 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:40.821170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:40201 (local address 127.0.203.62:44551)
0620 19:08:40.821241 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:40.821243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:40.821464 (+ 221us) server_negotiation.cc:407] Connection header received
0620 19:08:40.822017 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:40.822019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:40.822033 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:40.822084 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:40.822754 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:40.823344 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:40.824324 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:40.824527 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:40.824613 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:40.825507 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:40.825509 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:40.825509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:40.825515 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:40.825537 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:40.825541 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:40.825603 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:40.825719 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 19:08:40.825754 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:08:40.826045 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:40.821168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:44551 (local address 127.0.203.1:40201)
0620 19:08:40.821346 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:40.821380 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:40.821900 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:40.822153 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:40.822157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:40.822167 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:40.822640 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:40.822644 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:40.823437 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:40.823438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:40.824226 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:40.824228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:40.824622 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:40.824623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:40.824763 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:40.825360 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:08:40.825360 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:40.825390 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:08:40.825412 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:40.825604 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:40.825617 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:40.825620 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:08:40.825721 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 19:08:40.825825 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260620 19:08:40.826709 10226 heartbeater.cc:347] Connected to a master server at 127.0.203.62:44551
I20260620 19:08:40.826836 10226 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:40.827026 10226 heartbeater.cc:511] Master 127.0.203.62:44551 requested a full tablet report, sending...
I20260620 19:08:40.827559 10038 ts_manager.cc:194] Registered new tserver with Master: 4a89ee16b97a49998f8526d41e103cb1 (127.0.203.1:40385)
I20260620 19:08:40.828203 10038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.1:40201
I20260620 19:08:40.828982 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 10098
I20260620 19:08:40.829078 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:08:40.829294 812 external_mini_cluster.cc:1428] Setting key b6921fcaec759ca3ca2886a3ea650524
I20260620 19:08:40.830123 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44551
--builtin_ntp_servers=127.0.203.20:39433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 19:08:40.911067 10229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:40.911347 10229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:40.911369 10229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:40.912557 10229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:40.912598 10229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:40.912654 10229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:08:40.914049 10229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:39433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:40.914299 10229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:40.914567 10229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:40.919636 10235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:40.919652 10237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:40.919656 10234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:40.919890 10229 server_base.cc:1034] running on GCE node
I20260620 19:08:40.920085 10229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:40.920292 10229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:40.921432 10229 hybrid_clock.cc:648] HybridClock initialized: now 1781982520921414 us; error 34 us; skew 500 ppm
I20260620 19:08:40.922626 10229 webserver.cc:460] Webserver started at http://127.0.203.2:34443/ using document root <none> and password file <none>
I20260620 19:08:40.922878 10229 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:40.922925 10229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:40.923092 10229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:40.924173 10229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "e7305d349d704f2e8723bb84aaa0772e"
format_stamp: "Formatted at 2026-06-20 19:08:40 on dist-test-slave-wd30"
server_key: "53e0e8c82ca44279749dc71812c68c4e"
server_key_iv: "619f25e9bec04499aad8178efa924c5b"
server_key_version: "encryptionkey@0"
I20260620 19:08:40.924540 10229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "e7305d349d704f2e8723bb84aaa0772e"
format_stamp: "Formatted at 2026-06-20 19:08:40 on dist-test-slave-wd30"
server_key: "53e0e8c82ca44279749dc71812c68c4e"
server_key_iv: "619f25e9bec04499aad8178efa924c5b"
server_key_version: "encryptionkey@0"
I20260620 19:08:40.925913 10229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:08:40.926724 10243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:40.926918 10229 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:08:40.927018 10229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "e7305d349d704f2e8723bb84aaa0772e"
format_stamp: "Formatted at 2026-06-20 19:08:40 on dist-test-slave-wd30"
server_key: "53e0e8c82ca44279749dc71812c68c4e"
server_key_iv: "619f25e9bec04499aad8178efa924c5b"
server_key_version: "encryptionkey@0"
I20260620 19:08:40.927098 10229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:40.949235 10229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:40.949585 10229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:40.949745 10229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:40.949985 10229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:40.950335 10229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:40.950394 10229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:40.950436 10229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:40.950465 10229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:40.956490 10229 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:35009
I20260620 19:08:40.956535 10356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:35009 every 8 connection(s)
I20260620 19:08:40.956868 10229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:08:40.961776 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:40.957614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:59707 (local address 127.0.203.62:44551)
0620 19:08:40.957661 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:40.957663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:40.957907 (+ 244us) server_negotiation.cc:407] Connection header received
0620 19:08:40.958380 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:40.958381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:40.958392 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:40.958456 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:40.959105 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:40.959619 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:40.960558 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:40.960739 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:40.960791 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:40.961548 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:40.961549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:40.961549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:40.961553 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:40.961567 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:40.961571 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:40.961613 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:40.961691 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 19:08:40.961717 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:08:40.961963 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:40.957576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:44551 (local address 127.0.203.2:59707)
0620 19:08:40.957779 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:40.957828 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:40.958299 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:40.958525 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:40.958529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:40.958538 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:40.959033 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:40.959037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:40.959709 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:40.959710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:40.960482 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:40.960484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:40.960811 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:40.960812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:40.960917 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:40.961420 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:08:40.961421 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:40.961449 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:08:40.961470 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:40.961624 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:40.961628 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:40.961630 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:08:40.961681 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:08:40.961764 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
I20260620 19:08:40.962494 10357 heartbeater.cc:347] Connected to a master server at 127.0.203.62:44551
I20260620 19:08:40.962601 10357 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:40.962790 10357 heartbeater.cc:511] Master 127.0.203.62:44551 requested a full tablet report, sending...
I20260620 19:08:40.963207 10038 ts_manager.cc:194] Registered new tserver with Master: e7305d349d704f2e8723bb84aaa0772e (127.0.203.2:35009)
I20260620 19:08:40.963584 10038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.2:59707
I20260620 19:08:40.965013 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 10229
I20260620 19:08:40.965085 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:08:40.965296 812 external_mini_cluster.cc:1428] Setting key 79cac2e2068e68535eb7ed3238eca664
I20260620 19:08:40.966164 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44551
--builtin_ntp_servers=127.0.203.20:39433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 19:08:41.047089 10360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:41.047335 10360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:41.047358 10360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:41.048461 10360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:41.048494 10360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:41.048518 10360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:08:41.049754 10360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:39433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:41.049937 10360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:41.050180 10360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:41.055145 10365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:41.055174 10366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:41.055145 10368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:41.055380 10360 server_base.cc:1034] running on GCE node
I20260620 19:08:41.055605 10360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:41.055893 10360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:41.057054 10360 hybrid_clock.cc:648] HybridClock initialized: now 1781982521057036 us; error 29 us; skew 500 ppm
I20260620 19:08:41.058355 10360 webserver.cc:460] Webserver started at http://127.0.203.3:36273/ using document root <none> and password file <none>
I20260620 19:08:41.058602 10360 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:41.058676 10360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:41.058820 10360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:41.059966 10360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "33c8ed1a79e54487bd9baebd017bc2a7"
format_stamp: "Formatted at 2026-06-20 19:08:41 on dist-test-slave-wd30"
server_key: "7bce1c9602dcf1547543383ae4c52a69"
server_key_iv: "56edec701715a67e989cc774b48d903a"
server_key_version: "encryptionkey@0"
I20260620 19:08:41.060364 10360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "33c8ed1a79e54487bd9baebd017bc2a7"
format_stamp: "Formatted at 2026-06-20 19:08:41 on dist-test-slave-wd30"
server_key: "7bce1c9602dcf1547543383ae4c52a69"
server_key_iv: "56edec701715a67e989cc774b48d903a"
server_key_version: "encryptionkey@0"
I20260620 19:08:41.061640 10360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:08:41.062305 10374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:41.062465 10360 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:08:41.062572 10360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "33c8ed1a79e54487bd9baebd017bc2a7"
format_stamp: "Formatted at 2026-06-20 19:08:41 on dist-test-slave-wd30"
server_key: "7bce1c9602dcf1547543383ae4c52a69"
server_key_iv: "56edec701715a67e989cc774b48d903a"
server_key_version: "encryptionkey@0"
I20260620 19:08:41.062646 10360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:41.084506 10360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:41.084862 10360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:41.085031 10360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:41.085280 10360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:41.085620 10360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:41.085675 10360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:41.085717 10360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:41.085772 10360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:41.091362 10360 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:42947
I20260620 19:08:41.091405 10487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:42947 every 8 connection(s)
I20260620 19:08:41.091722 10360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:08:41.097103 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:41.092464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:49139 (local address 127.0.203.62:44551)
0620 19:08:41.092516 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:41.092517 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:41.092764 (+ 247us) server_negotiation.cc:407] Connection header received
0620 19:08:41.093295 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:41.093296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:41.093309 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:41.093362 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:41.094060 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:41.094557 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:41.095636 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:41.095835 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:41.095915 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:41.096802 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:41.096804 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:41.096804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:41.096808 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:41.096824 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:41.096830 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:41.096898 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:41.096991 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 19:08:41.097020 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:08:41.097347 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:41.092447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:44551 (local address 127.0.203.3:49139)
0620 19:08:41.092635 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:41.092669 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:41.093191 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:41.093444 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:41.093449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:41.093465 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:41.093963 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:41.093967 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:41.094658 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:41.094659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:41.095533 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:41.095536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:41.095933 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:41.095934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:41.096065 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:41.096637 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:08:41.096637 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:41.096666 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:08:41.096693 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:41.096897 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:41.096904 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:41.096907 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:08:41.096996 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 19:08:41.097096 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260620 19:08:41.097769 10488 heartbeater.cc:347] Connected to a master server at 127.0.203.62:44551
I20260620 19:08:41.097854 10488 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:41.098065 10488 heartbeater.cc:511] Master 127.0.203.62:44551 requested a full tablet report, sending...
I20260620 19:08:41.098415 10038 ts_manager.cc:194] Registered new tserver with Master: 33c8ed1a79e54487bd9baebd017bc2a7 (127.0.203.3:42947)
I20260620 19:08:41.098788 10038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.3:49139
I20260620 19:08:41.101064 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 10360
I20260620 19:08:41.101137 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:08:41.101354 812 external_mini_cluster.cc:1428] Setting key 51e436bc28f6db7e5f691210ceef0043
I20260620 19:08:41.102270 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:08:41.104108 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 10098
W20260620 19:08:41.104029 10079 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:41.103601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60946 (local address 127.0.203.62:44551)
0620 19:08:41.103660 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:41.103662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:41.103774 (+ 112us) server_negotiation.cc:407] Connection header received
0620 19:08:41.103815 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:41.103815 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:41.103825 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:41.103872 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:41.103984 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60946: BlockingRecv error: recv got EOF from 127.0.0.1:60946 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:08:41.109169 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 10229
I20260620 19:08:41.113212 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 10360
I20260620 19:08:41.117375 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 10007
2026-06-20T19:08:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (621 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T19:08:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:08:41Z Disabled control of system clock
I20260620 19:08:41.143219 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:41267
--webserver_interface=127.0.203.62
--webserver_port=0
--builtin_ntp_servers=127.0.203.20:42705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:41267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 19:08:41.227049 10499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:41.227324 10499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:41.227346 10499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:41.228487 10499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:41.228526 10499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:41.228538 10499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:41.228551 10499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:41.228564 10499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:08:41.229911 10499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:42705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:41267
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:41267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:41.230141 10499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:41.230397 10499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:41.235796 10504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:41.235798 10505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:41.235816 10499 server_base.cc:1034] running on GCE node
W20260620 19:08:41.235796 10507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:41.236297 10499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:41.236686 10499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:41.237910 10499 hybrid_clock.cc:648] HybridClock initialized: now 1781982521237862 us; error 59 us; skew 500 ppm
I20260620 19:08:41.239387 10499 webserver.cc:460] Webserver started at http://127.0.203.62:44643/ using document root <none> and password file <none>
I20260620 19:08:41.239641 10499 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:41.239711 10499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:41.239878 10499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:41.241024 10499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "1076f3d785ec4c578ad20a8258ec61fc"
format_stamp: "Formatted at 2026-06-20 19:08:41 on dist-test-slave-wd30"
server_key: "bee0a9335d7cd70c3b47f7075d4c254c"
server_key_iv: "9f7da91329f3621730fb4bde885d25d7"
server_key_version: "encryptionkey@0"
I20260620 19:08:41.241410 10499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "1076f3d785ec4c578ad20a8258ec61fc"
format_stamp: "Formatted at 2026-06-20 19:08:41 on dist-test-slave-wd30"
server_key: "bee0a9335d7cd70c3b47f7075d4c254c"
server_key_iv: "9f7da91329f3621730fb4bde885d25d7"
server_key_version: "encryptionkey@0"
I20260620 19:08:41.242730 10499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:08:41.243637 10513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:41.243865 10499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 19:08:41.243986 10499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "1076f3d785ec4c578ad20a8258ec61fc"
format_stamp: "Formatted at 2026-06-20 19:08:41 on dist-test-slave-wd30"
server_key: "bee0a9335d7cd70c3b47f7075d4c254c"
server_key_iv: "9f7da91329f3621730fb4bde885d25d7"
server_key_version: "encryptionkey@0"
I20260620 19:08:41.244074 10499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:41.247539 10499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:41.247807 10499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:41.247951 10499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:41.251848 10499 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:41267
I20260620 19:08:41.251932 10565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:41267 every 8 connection(s)
I20260620 19:08:41.252291 10499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:41.252995 10566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:41.255815 10566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc: Bootstrap starting.
I20260620 19:08:41.256428 10566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:41.256735 10566 log.cc:824] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:41.257328 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 10499
I20260620 19:08:41.257421 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:08:41.257515 10566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc: No bootstrap required, opened a new log
I20260620 19:08:41.257617 812 external_mini_cluster.cc:1428] Setting key 94ca83197756fd26116ddd2d77660f66
I20260620 19:08:41.259922 10566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1076f3d785ec4c578ad20a8258ec61fc" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41267 } }
I20260620 19:08:41.260092 10566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:41.260141 10566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1076f3d785ec4c578ad20a8258ec61fc, State: Initialized, Role: FOLLOWER
I20260620 19:08:41.260315 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:41.258157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39202 (local address 127.0.203.62:41267)
0620 19:08:41.258397 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:41.258404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:41.258429 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:08:41.258858 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:41.258864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:41.258903 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:41.259020 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:41.259766 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:41.259769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:41.259770 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:41.259774 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:41.259826 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:41.259836 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:41.259915 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:41.259989 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 19:08:41.260050 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260620 19:08:41.260329 10566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1076f3d785ec4c578ad20a8258ec61fc" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41267 } }
I20260620 19:08:41.260490 10566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:41.260520 10566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:41.260566 10566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:41.261543 10566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1076f3d785ec4c578ad20a8258ec61fc" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41267 } }
I20260620 19:08:41.261632 10566 leader_election.cc:302] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1076f3d785ec4c578ad20a8258ec61fc; no voters:
I20260620 19:08:41.261832 10566 leader_election.cc:288] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:08:41.261911 10571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:41.262068 10571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc [term 1 LEADER]: Becoming Leader. State: Replica: 1076f3d785ec4c578ad20a8258ec61fc, State: Running, Role: LEADER
I20260620 19:08:41.262100 10566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:41.262171 10571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1076f3d785ec4c578ad20a8258ec61fc" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41267 } }
I20260620 19:08:41.262564 10573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1076f3d785ec4c578ad20a8258ec61fc. Latest consensus state: current_term: 1 leader_uuid: "1076f3d785ec4c578ad20a8258ec61fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1076f3d785ec4c578ad20a8258ec61fc" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41267 } } }
I20260620 19:08:41.262683 10573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:41.262552 10572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1076f3d785ec4c578ad20a8258ec61fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1076f3d785ec4c578ad20a8258ec61fc" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41267 } } }
I20260620 19:08:41.262859 10572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:41.263183 10579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:08:41.263662 10579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:08:41.265206 10579 catalog_manager.cc:1347] Generated new cluster ID: 1c4b9718a4bb4603ae40a91b9bad563a
I20260620 19:08:41.265261 10579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:08:41.299698 10579 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:08:41.299877 10579 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:41.305660 10579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1076f3d785ec4c578ad20a8258ec61fc: Generated new TSK 0
I20260620 19:08:41.305842 10579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:08:41.316767 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:41267
--builtin_ntp_servers=127.0.203.20:42705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 19:08:41.400727 10590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:41.400938 10590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:41.400961 10590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:41.402221 10590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:41.402261 10590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:41.402287 10590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:08:41.403702 10590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:42705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:41267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:41.403957 10590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:41.404259 10590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:41.409554 10598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:41.409546 10595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:41.409804 10590 server_base.cc:1034] running on GCE node
W20260620 19:08:41.409859 10596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:41.410107 10590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:41.410374 10590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:41.413695 10590 hybrid_clock.cc:648] HybridClock initialized: now 1781982521413647 us; error 58 us; skew 500 ppm
I20260620 19:08:41.418869 10590 webserver.cc:460] Webserver started at http://127.0.203.1:36895/ using document root <none> and password file <none>
I20260620 19:08:41.419087 10590 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:41.419214 10590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:41.419327 10590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:41.420382 10590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "cf1ed1fc27b84454b094e8c388eae1ef"
format_stamp: "Formatted at 2026-06-20 19:08:41 on dist-test-slave-wd30"
server_key: "1c258587ab290e8cb0c9aa4ec7de46e0"
server_key_iv: "7b79e5cd2a85783b3ce913c094fead40"
server_key_version: "encryptionkey@0"
I20260620 19:08:41.420687 10590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "cf1ed1fc27b84454b094e8c388eae1ef"
format_stamp: "Formatted at 2026-06-20 19:08:41 on dist-test-slave-wd30"
server_key: "1c258587ab290e8cb0c9aa4ec7de46e0"
server_key_iv: "7b79e5cd2a85783b3ce913c094fead40"
server_key_version: "encryptionkey@0"
I20260620 19:08:41.421983 10590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:08:41.422786 10604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:41.422997 10590 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:08:41.423071 10590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "cf1ed1fc27b84454b094e8c388eae1ef"
format_stamp: "Formatted at 2026-06-20 19:08:41 on dist-test-slave-wd30"
server_key: "1c258587ab290e8cb0c9aa4ec7de46e0"
server_key_iv: "7b79e5cd2a85783b3ce913c094fead40"
server_key_version: "encryptionkey@0"
I20260620 19:08:41.423206 10590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:41.426540 10590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:41.426792 10590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:41.426889 10590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:41.427054 10590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:41.427385 10590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:41.427419 10590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:41.427440 10590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:41.427453 10590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:41.433311 10590 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:35061
I20260620 19:08:41.433367 10717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:35061 every 8 connection(s)
I20260620 19:08:41.433707 10590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:08:41.436831 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:41.434466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:47153 (local address 127.0.203.62:41267)
0620 19:08:41.434511 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:41.434513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:41.434764 (+ 251us) server_negotiation.cc:407] Connection header received
0620 19:08:41.435301 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:41.435302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:41.435307 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:41.435365 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:41.436578 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:41.436580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:41.436580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:41.436585 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:41.436601 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:41.436605 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:41.436688 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:41.436760 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 19:08:41.436774 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:08:41.437045 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:41.434441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:41267 (local address 127.0.203.1:47153)
0620 19:08:41.434628 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:41.434678 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:41.435210 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:41.435426 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:41.435429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:41.435437 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:41.436407 (+ 970us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:08:41.436408 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:41.436437 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:08:41.436466 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:41.436688 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:41.436690 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:41.436692 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:08:41.436744 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 19:08:41.436794 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260620 19:08:41.437651 10718 heartbeater.cc:347] Connected to a master server at 127.0.203.62:41267
I20260620 19:08:41.437763 10718 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:41.437937 10718 heartbeater.cc:511] Master 127.0.203.62:41267 requested a full tablet report, sending...
I20260620 19:08:41.438429 10530 ts_manager.cc:194] Registered new tserver with Master: cf1ed1fc27b84454b094e8c388eae1ef (127.0.203.1:35061)
I20260620 19:08:41.441932 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 10590
I20260620 19:08:41.442021 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:08:41.442407 812 external_mini_cluster.cc:1428] Setting key 360fafad810324a69ae38064edf46cca
I20260620 19:08:41.443449 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:41267
--builtin_ntp_servers=127.0.203.20:42705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 19:08:41.526132 10721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:41.526412 10721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:41.526454 10721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:41.527808 10721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:41.527891 10721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:41.527940 10721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:08:41.529347 10721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:42705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:41267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:41.529634 10721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:41.529963 10721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:41.535234 10727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:41.535282 10726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:41.535245 10729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:41.535315 10721 server_base.cc:1034] running on GCE node
I20260620 19:08:41.535625 10721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:41.535862 10721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:41.537019 10721 hybrid_clock.cc:648] HybridClock initialized: now 1781982521537013 us; error 41 us; skew 500 ppm
I20260620 19:08:41.538338 10721 webserver.cc:460] Webserver started at http://127.0.203.2:34081/ using document root <none> and password file <none>
I20260620 19:08:41.538578 10721 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:41.538642 10721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:41.538791 10721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:41.539909 10721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "ad9c70bfb4b940dfbf25543b600d16e0"
format_stamp: "Formatted at 2026-06-20 19:08:41 on dist-test-slave-wd30"
server_key: "ba8c4e36cdbc0c4892bd97191c655c6a"
server_key_iv: "a01bfeba45197e6285d1806e39ac9f92"
server_key_version: "encryptionkey@0"
I20260620 19:08:41.540272 10721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "ad9c70bfb4b940dfbf25543b600d16e0"
format_stamp: "Formatted at 2026-06-20 19:08:41 on dist-test-slave-wd30"
server_key: "ba8c4e36cdbc0c4892bd97191c655c6a"
server_key_iv: "a01bfeba45197e6285d1806e39ac9f92"
server_key_version: "encryptionkey@0"
I20260620 19:08:41.541604 10721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:08:41.542471 10735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:41.542661 10721 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:08:41.542766 10721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "ad9c70bfb4b940dfbf25543b600d16e0"
format_stamp: "Formatted at 2026-06-20 19:08:41 on dist-test-slave-wd30"
server_key: "ba8c4e36cdbc0c4892bd97191c655c6a"
server_key_iv: "a01bfeba45197e6285d1806e39ac9f92"
server_key_version: "encryptionkey@0"
I20260620 19:08:41.542858 10721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:41.546258 10721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:41.546538 10721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:41.546681 10721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:41.546893 10721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:41.547248 10721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:41.547300 10721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:41.547348 10721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:41.547377 10721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:41.553068 10721 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:45257
I20260620 19:08:41.553103 10848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:45257 every 8 connection(s)
I20260620 19:08:41.553442 10721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:08:41.556309 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:41.554187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:36879 (local address 127.0.203.62:41267)
0620 19:08:41.554231 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:41.554233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:41.554472 (+ 239us) server_negotiation.cc:407] Connection header received
0620 19:08:41.554920 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:41.554921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:41.554926 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:41.554989 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:41.556121 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:41.556122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:41.556122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:41.556126 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:41.556141 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:41.556145 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:41.556192 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:41.556238 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 19:08:41.556253 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:08:41.556548 10851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:41.554156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:41267 (local address 127.0.203.2:36879)
0620 19:08:41.554345 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:41.554386 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:41.554839 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:41.555043 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:41.555046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:41.555054 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:41.555979 (+ 925us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:08:41.555980 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:41.556008 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:08:41.556036 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:41.556190 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:41.556192 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:41.556194 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:08:41.556244 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 19:08:41.556290 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260620 19:08:41.557045 10849 heartbeater.cc:347] Connected to a master server at 127.0.203.62:41267
I20260620 19:08:41.557122 10849 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:41.557235 10849 heartbeater.cc:511] Master 127.0.203.62:41267 requested a full tablet report, sending...
I20260620 19:08:41.557549 10530 ts_manager.cc:194] Registered new tserver with Master: ad9c70bfb4b940dfbf25543b600d16e0 (127.0.203.2:45257)
I20260620 19:08:41.558161 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 10721
I20260620 19:08:41.558264 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:08:41.558463 812 external_mini_cluster.cc:1428] Setting key 90a6641ce7962662b897bd33364f7640
I20260620 19:08:41.559377 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:41267
--builtin_ntp_servers=127.0.203.20:42705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 19:08:41.642038 10852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:41.642244 10852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:41.642267 10852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:41.643399 10852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:41.643432 10852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:41.643457 10852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:08:41.644728 10852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:42705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:41267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:41.644933 10852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:41.645255 10852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:41.650513 10858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:41.650485 10857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:41.650494 10860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:41.650553 10852 server_base.cc:1034] running on GCE node
I20260620 19:08:41.650840 10852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:41.651083 10852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:41.652271 10852 hybrid_clock.cc:648] HybridClock initialized: now 1781982521652254 us; error 32 us; skew 500 ppm
I20260620 19:08:41.653563 10852 webserver.cc:460] Webserver started at http://127.0.203.3:41505/ using document root <none> and password file <none>
I20260620 19:08:41.653828 10852 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:41.653903 10852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:41.654055 10852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:41.655283 10852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "5643536603b94d2eac92f0a12c648674"
format_stamp: "Formatted at 2026-06-20 19:08:41 on dist-test-slave-wd30"
server_key: "a28857badced7d88eee06a3ec077f508"
server_key_iv: "6df456cb7c48f893bc1ca4ecce30abdd"
server_key_version: "encryptionkey@0"
I20260620 19:08:41.655762 10852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "5643536603b94d2eac92f0a12c648674"
format_stamp: "Formatted at 2026-06-20 19:08:41 on dist-test-slave-wd30"
server_key: "a28857badced7d88eee06a3ec077f508"
server_key_iv: "6df456cb7c48f893bc1ca4ecce30abdd"
server_key_version: "encryptionkey@0"
I20260620 19:08:41.657207 10852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:41.658069 10866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:41.658272 10852 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:41.658373 10852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "5643536603b94d2eac92f0a12c648674"
format_stamp: "Formatted at 2026-06-20 19:08:41 on dist-test-slave-wd30"
server_key: "a28857badced7d88eee06a3ec077f508"
server_key_iv: "6df456cb7c48f893bc1ca4ecce30abdd"
server_key_version: "encryptionkey@0"
I20260620 19:08:41.658463 10852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:41.661957 10852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:41.662268 10852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:41.662407 10852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:41.662626 10852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:41.663017 10852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:41.663059 10852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:41.663091 10852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:41.663144 10852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:41.668752 10852 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:45725
I20260620 19:08:41.668823 10979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:45725 every 8 connection(s)
I20260620 19:08:41.669174 10852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:08:41.672232 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:41.670014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:40119 (local address 127.0.203.62:41267)
0620 19:08:41.670106 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:41.670108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:41.670213 (+ 105us) server_negotiation.cc:407] Connection header received
0620 19:08:41.670708 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:41.670709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:41.670714 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:41.670770 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:41.672028 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:41.672029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:41.672029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:08:41.672033 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:41.672049 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:08:41.672053 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:41.672106 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:41.672165 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 19:08:41.672177 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 19:08:41.672428 10982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:41.669920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:41267 (local address 127.0.203.3:40119)
0620 19:08:41.670108 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:41.670143 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:41.670607 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:41.670848 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:41.670851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:41.670860 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:41.671848 (+ 988us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:08:41.671848 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:41.671879 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:08:41.671922 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:41.672105 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:41.672107 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:41.672110 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:08:41.672162 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 19:08:41.672213 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260620 19:08:41.672994 10980 heartbeater.cc:347] Connected to a master server at 127.0.203.62:41267
I20260620 19:08:41.673076 10980 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:41.673255 10980 heartbeater.cc:511] Master 127.0.203.62:41267 requested a full tablet report, sending...
I20260620 19:08:41.673643 10530 ts_manager.cc:194] Registered new tserver with Master: 5643536603b94d2eac92f0a12c648674 (127.0.203.3:45725)
I20260620 19:08:41.674117 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 10852
I20260620 19:08:41.674222 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:08:41.674415 812 external_mini_cluster.cc:1428] Setting key 88a27d90f6c757a2c4ca4014ea5ddf22
I20260620 19:08:41.675336 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 19:08:41.677320 10570 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:08:41.676712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39204 (local address 127.0.203.62:41267)
0620 19:08:41.676814 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:41.676816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:41.676920 (+ 104us) server_negotiation.cc:407] Connection header received
0620 19:08:41.676995 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:41.676996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:41.677001 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:41.677059 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:41.677228 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39204: BlockingRecv error: recv got EOF from 127.0.0.1:39204 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 19:08:41.677366 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 10590
I20260620 19:08:41.682245 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 10721
I20260620 19:08:41.686578 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 10852
I20260620 19:08:41.691094 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 10499
2026-06-20T19:08:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (574 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:08:41 dist-test-slave-wd30 krb5kdc[10990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:08:41 dist-test-slave-wd30 krb5kdc[10990](info): set up 2 sockets
Jun 20 19:08:41 dist-test-slave-wd30 krb5kdc[10990](info): commencing operation
krb5kdc: starting...
W20260620 19:08:42.748227 812 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:08:42 dist-test-slave-wd30 krb5kdc[10990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:08:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:08:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:08:42.868108 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:35747
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:35747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:42.949095 11003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:42.949312 11003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:42.949335 11003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:42.950551 11003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:08:42.950596 11003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:42.950611 11003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:42.950623 11003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:08:42.950681 11003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:08:42.952260 11003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:35747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:35747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:42.952548 11003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:42.952811 11003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:42.957795 11009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:42.957782 11008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:42.957782 11011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:42.958034 11003 server_base.cc:1034] running on GCE node
I20260620 19:08:42.958309 11003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:42.958599 11003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:42.959812 11003 hybrid_clock.cc:648] HybridClock initialized: now 1781982522959787 us; error 42 us; skew 500 ppm
Jun 20 19:08:42 dist-test-slave-wd30 krb5kdc[10990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:42.962389 11003 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:08:42.962908 11003 webserver.cc:460] Webserver started at http://127.0.203.62:36751/ using document root <none> and password file <none>
I20260620 19:08:42.963204 11003 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:42.963275 11003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:42.963440 11003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:42.964535 11003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "910275ae92764f408ab27d278f37861a"
format_stamp: "Formatted at 2026-06-20 19:08:42 on dist-test-slave-wd30"
server_key: "1eb5ef9855534852a222ba53739edf74"
server_key_iv: "7822d8673d8cfd507d88d680e018a237"
server_key_version: "encryptionkey@0"
I20260620 19:08:42.964913 11003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "910275ae92764f408ab27d278f37861a"
format_stamp: "Formatted at 2026-06-20 19:08:42 on dist-test-slave-wd30"
server_key: "1eb5ef9855534852a222ba53739edf74"
server_key_iv: "7822d8673d8cfd507d88d680e018a237"
server_key_version: "encryptionkey@0"
I20260620 19:08:42.966264 11003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:08:42.967046 11018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:42.967283 11003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:42.967408 11003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "910275ae92764f408ab27d278f37861a"
format_stamp: "Formatted at 2026-06-20 19:08:42 on dist-test-slave-wd30"
server_key: "1eb5ef9855534852a222ba53739edf74"
server_key_iv: "7822d8673d8cfd507d88d680e018a237"
server_key_version: "encryptionkey@0"
I20260620 19:08:42.967507 11003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:42.978291 11003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:42.980502 11003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:42.980675 11003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:42.984769 11003 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:35747
I20260620 19:08:42.984855 11070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:35747 every 8 connection(s)
I20260620 19:08:42.985217 11003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:08:42.986232 11071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:42.988539 11071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a: Bootstrap starting.
I20260620 19:08:42.989130 11071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:42.989364 11071 log.cc:824] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:42.990046 11071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a: No bootstrap required, opened a new log
I20260620 19:08:42.991684 11071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910275ae92764f408ab27d278f37861a" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35747 } }
I20260620 19:08:42.991824 11071 raft_consensus.cc:374] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:42.991860 11071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910275ae92764f408ab27d278f37861a, State: Initialized, Role: FOLLOWER
I20260620 19:08:42.992005 11071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910275ae92764f408ab27d278f37861a" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35747 } }
I20260620 19:08:42.992097 11071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:08:42.992121 11071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:08:42.992151 11071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:42.992903 11071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910275ae92764f408ab27d278f37861a" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35747 } }
I20260620 19:08:42.992990 11071 leader_election.cc:302] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 910275ae92764f408ab27d278f37861a; no voters:
I20260620 19:08:42.993141 11071 leader_election.cc:288] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:08:42.993206 11074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:42.993294 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 11003
I20260620 19:08:42.993346 11074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a [term 1 LEADER]: Becoming Leader. State: Replica: 910275ae92764f408ab27d278f37861a, State: Running, Role: LEADER
I20260620 19:08:42.993402 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:08:42.993436 11071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:08:42.993474 11074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910275ae92764f408ab27d278f37861a" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35747 } }
I20260620 19:08:42.993585 812 external_mini_cluster.cc:1428] Setting key 349fc5b27f7962788808907959b4f55e
I20260620 19:08:42.993842 11076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "910275ae92764f408ab27d278f37861a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910275ae92764f408ab27d278f37861a" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35747 } } }
I20260620 19:08:42.993846 11074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 910275ae92764f408ab27d278f37861a. Latest consensus state: current_term: 1 leader_uuid: "910275ae92764f408ab27d278f37861a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910275ae92764f408ab27d278f37861a" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35747 } } }
I20260620 19:08:42.993945 11076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:42.993955 11074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a [sys.catalog]: This master's current role is: LEADER
I20260620 19:08:42.994611 11085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:08:42.995115 11085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:08:42.996748 11085 catalog_manager.cc:1347] Generated new cluster ID: 3d1d1be78c52411d88f1415dd9d3bad1
I20260620 19:08:42.996800 11085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:08:42 dist-test-slave-wd30 krb5kdc[10990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:43.004122 11085 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:08:43.004416 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:42.994099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.0.203.62:35747)
0620 19:08:42.994420 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:42.994431 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:42.994627 (+ 196us) server_negotiation.cc:407] Connection header received
0620 19:08:42.994910 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:42.994919 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:42.994982 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:42.995157 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:42.995872 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:42.997093 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:42.997821 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:42.998257 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:42.998398 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:43.000847 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:43.000851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:43.000851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:43.000863 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:43.002584 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:43.002965 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:43.002968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:43.002968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:43.003071 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:43.003310 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:43.003311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:43.003312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:43.003561 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:43.003770 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:43.004019 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 19:08:43.004172 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":72,"threads_started":1}
I20260620 19:08:43.004848 11085 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:08:43.010926 11085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a: Generated new TSK 0
I20260620 19:08:43.011098 11085 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
I20260620 19:08:43.045404 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:35747
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:43.140805 11097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:43.141012 11097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:43.141037 11097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:43.142215 11097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:43.142261 11097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:43.142287 11097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:08:43.143846 11097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:35747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:43.144084 11097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:43.144332 11097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:43.149657 11102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:43.149690 11103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:43.149780 11097 server_base.cc:1034] running on GCE node
W20260620 19:08:43.149665 11105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:43.150104 11097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:43.150349 11097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:43.151535 11097 hybrid_clock.cc:648] HybridClock initialized: now 1781982523151522 us; error 30 us; skew 500 ppm
Jun 20 19:08:43 dist-test-slave-wd30 krb5kdc[10990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:43.154088 11097 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:08:43.154584 11097 webserver.cc:460] Webserver started at http://127.0.203.1:41187/ using document root <none> and password file <none>
I20260620 19:08:43.154810 11097 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:43.154877 11097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:43.155035 11097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:43.156162 11097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "ac0cf2efc19c495faa2f1834b957ebaa"
format_stamp: "Formatted at 2026-06-20 19:08:43 on dist-test-slave-wd30"
server_key: "4e1e310b566a58df2674b499dea9e935"
server_key_iv: "a38bb2f9b85628e238ae46695bd0a447"
server_key_version: "encryptionkey@0"
I20260620 19:08:43.156539 11097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "ac0cf2efc19c495faa2f1834b957ebaa"
format_stamp: "Formatted at 2026-06-20 19:08:43 on dist-test-slave-wd30"
server_key: "4e1e310b566a58df2674b499dea9e935"
server_key_iv: "a38bb2f9b85628e238ae46695bd0a447"
server_key_version: "encryptionkey@0"
I20260620 19:08:43.157881 11097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:08:43.158663 11112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:43.158880 11097 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:43.159005 11097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "ac0cf2efc19c495faa2f1834b957ebaa"
format_stamp: "Formatted at 2026-06-20 19:08:43 on dist-test-slave-wd30"
server_key: "4e1e310b566a58df2674b499dea9e935"
server_key_iv: "a38bb2f9b85628e238ae46695bd0a447"
server_key_version: "encryptionkey@0"
I20260620 19:08:43.159085 11097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:43.169479 11097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:43.171973 11097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:43.172149 11097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:43.172371 11097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:43.172693 11097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:43.172726 11097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:43.172749 11097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:43.172794 11097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:43.178325 11097 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:43941
I20260620 19:08:43.178421 11225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:43941 every 8 connection(s)
I20260620 19:08:43.178756 11097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:08:43.181064 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 11097
I20260620 19:08:43.181166 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:08:43.181363 812 external_mini_cluster.cc:1428] Setting key 64341b217c4072f50c5e9eb3f483c31f
Jun 20 19:08:43 dist-test-slave-wd30 krb5kdc[10990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.203.2@KRBTEST.COM; defaulting to no policy
I20260620 19:08:43.189666 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:43.179531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:37833 (local address 127.0.203.62:35747)
0620 19:08:43.179597 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:43.179599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:43.179791 (+ 192us) server_negotiation.cc:407] Connection header received
0620 19:08:43.180263 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:43.180264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:43.180275 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:43.180334 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:43.181568 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.182191 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.183821 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.184167 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.184251 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:43.186677 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:43.186684 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:43.186684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:43.186694 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:43.188211 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:43.188656 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:43.188657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:43.188658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:43.188718 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:43.188981 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:43.188982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:43.188982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:43.189101 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:43.189179 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:43.189463 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 19:08:43.189571 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:08:43.189826 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:43.179478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:35747 (local address 127.0.203.1:37833)
0620 19:08:43.179654 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:43.179698 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:43.180178 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:43.180388 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:43.180391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:43.181076 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:43.181383 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:43.181390 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.182292 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.182293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:43.183717 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:43.183720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.184278 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.184279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:43.184407 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:43.184998 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:43.185008 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:43.186563 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:43.188315 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:43.188323 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:43.188325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:43.188563 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:43.188789 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:43.188790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:43.188790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:43.188910 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:43.189170 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:43.189177 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:43.189323 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 19:08:43.189429 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 19:08:43.189591 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":104,"threads_started":1}
I20260620 19:08:43.190342 11226 heartbeater.cc:347] Connected to a master server at 127.0.203.62:35747
I20260620 19:08:43.190428 11226 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:43.190627 11226 heartbeater.cc:511] Master 127.0.203.62:35747 requested a full tablet report, sending...
I20260620 19:08:43.191210 11035 ts_manager.cc:194] Registered new tserver with Master: ac0cf2efc19c495faa2f1834b957ebaa (127.0.203.1:43941)
I20260620 19:08:43.191816 11035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:37833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
I20260620 19:08:43.206713 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:35747
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:43.289193 11231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:43.289417 11231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:43.289441 11231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:43.290634 11231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:43.290680 11231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:43.290706 11231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:08:43.292282 11231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:35747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:43.292536 11231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:43.292757 11231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:43.298036 11239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:43.298044 11237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:43.298020 11236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:43.298339 11231 server_base.cc:1034] running on GCE node
I20260620 19:08:43.298599 11231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:43.298856 11231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:43.300050 11231 hybrid_clock.cc:648] HybridClock initialized: now 1781982523300044 us; error 38 us; skew 500 ppm
Jun 20 19:08:43 dist-test-slave-wd30 krb5kdc[10990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:43.302740 11231 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:08:43.303313 11231 webserver.cc:460] Webserver started at http://127.0.203.2:46317/ using document root <none> and password file <none>
I20260620 19:08:43.303555 11231 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:43.303622 11231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:43.303759 11231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:43.304867 11231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "e4feb4b86005485eba0f8d2dd73b93c8"
format_stamp: "Formatted at 2026-06-20 19:08:43 on dist-test-slave-wd30"
server_key: "d523d7620d4475f0dc69550f56acc49e"
server_key_iv: "5a7fb741f8ffd95bc2f6f9d024144d71"
server_key_version: "encryptionkey@0"
I20260620 19:08:43.305246 11231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "e4feb4b86005485eba0f8d2dd73b93c8"
format_stamp: "Formatted at 2026-06-20 19:08:43 on dist-test-slave-wd30"
server_key: "d523d7620d4475f0dc69550f56acc49e"
server_key_iv: "5a7fb741f8ffd95bc2f6f9d024144d71"
server_key_version: "encryptionkey@0"
I20260620 19:08:43.306627 11231 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:08:43.307435 11246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:43.307610 11231 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:43.307708 11231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "e4feb4b86005485eba0f8d2dd73b93c8"
format_stamp: "Formatted at 2026-06-20 19:08:43 on dist-test-slave-wd30"
server_key: "d523d7620d4475f0dc69550f56acc49e"
server_key_iv: "5a7fb741f8ffd95bc2f6f9d024144d71"
server_key_version: "encryptionkey@0"
I20260620 19:08:43.307796 11231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:43.324673 11231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:43.326848 11231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:43.327037 11231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:43.327323 11231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:43.327651 11231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:43.327708 11231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:43.327754 11231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:43.327780 11231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:43.333771 11231 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:44375
I20260620 19:08:43.333809 11359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:44375 every 8 connection(s)
I20260620 19:08:43.334154 11231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:08:43 dist-test-slave-wd30 krb5kdc[10990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:43.341812 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 11231
I20260620 19:08:43.341928 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:08:43.342146 812 external_mini_cluster.cc:1428] Setting key ff09fd48276e5fdaf6437f257c86eeb4
I20260620 19:08:43.343958 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:43.334957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:52507 (local address 127.0.203.62:35747)
0620 19:08:43.335100 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:43.335102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:43.335255 (+ 153us) server_negotiation.cc:407] Connection header received
0620 19:08:43.335773 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:43.335774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:43.335785 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:43.335858 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:43.336787 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.337421 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.338505 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.338722 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.338785 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:43.340878 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:43.340879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:43.340879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:43.340885 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:43.342491 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:43.342913 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:43.342914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:43.342914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:43.342981 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:43.343299 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:43.343300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:43.343301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:43.343421 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:43.343498 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:43.343785 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 19:08:43.343871 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 19:08:43.344172 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:43.334927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:35747 (local address 127.0.203.2:52507)
0620 19:08:43.335084 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:43.335153 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:43.335645 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:43.335921 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:43.335924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:43.336484 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:43.336678 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:43.336683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.337587 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.337588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:43.338408 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:43.338411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.338801 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.338802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:43.338911 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:43.339471 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:43.339480 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:43.340795 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:43.342550 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:43.342555 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:43.342557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:43.342829 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:43.343064 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:43.343064 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:43.343065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:43.343227 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:43.343489 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:43.343494 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:43.343644 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 19:08:43.343752 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 19:08:43.343909 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260620 19:08:43.344770 11360 heartbeater.cc:347] Connected to a master server at 127.0.203.62:35747
I20260620 19:08:43.344880 11360 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:43.345096 11360 heartbeater.cc:511] Master 127.0.203.62:35747 requested a full tablet report, sending...
I20260620 19:08:43.345475 11035 ts_manager.cc:194] Registered new tserver with Master: e4feb4b86005485eba0f8d2dd73b93c8 (127.0.203.2:44375)
I20260620 19:08:43.345984 11035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:52507
WARNING: no policy specified for kudu/127.0.203.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
I20260620 19:08:43.366431 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:35747
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:08:43.448311 11365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:08:43.448516 11365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:08:43.448549 11365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:08:43.449687 11365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:08:43.449729 11365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:08:43.449756 11365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:08:43.451305 11365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:35747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:08:43.451555 11365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:08:43.451786 11365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:08:43.456846 11373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:08:43.456902 11370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:43.456940 11365 server_base.cc:1034] running on GCE node
W20260620 19:08:43.456874 11371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:08:43.457194 11365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:08:43.457449 11365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:08:43.458621 11365 hybrid_clock.cc:648] HybridClock initialized: now 1781982523458618 us; error 37 us; skew 500 ppm
Jun 20 19:08:43 dist-test-slave-wd30 krb5kdc[10990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:08:43.461130 11365 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:08:43.461552 11365 webserver.cc:460] Webserver started at http://127.0.203.3:34889/ using document root <none> and password file <none>
I20260620 19:08:43.461794 11365 fs_manager.cc:359] Metadata directory not provided
I20260620 19:08:43.461861 11365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:08:43.462033 11365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:08:43.463101 11365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "ecdababd7a504f1e98b48aef99a91bd2"
format_stamp: "Formatted at 2026-06-20 19:08:43 on dist-test-slave-wd30"
server_key: "64583fd5c7a53e42cf1d6c29c6823451"
server_key_iv: "5e14c8e99970c518391d964fc24e401d"
server_key_version: "encryptionkey@0"
I20260620 19:08:43.463560 11365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "ecdababd7a504f1e98b48aef99a91bd2"
format_stamp: "Formatted at 2026-06-20 19:08:43 on dist-test-slave-wd30"
server_key: "64583fd5c7a53e42cf1d6c29c6823451"
server_key_iv: "5e14c8e99970c518391d964fc24e401d"
server_key_version: "encryptionkey@0"
I20260620 19:08:43.464879 11365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:08:43.465628 11380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:43.465801 11365 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:08:43.465910 11365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "ecdababd7a504f1e98b48aef99a91bd2"
format_stamp: "Formatted at 2026-06-20 19:08:43 on dist-test-slave-wd30"
server_key: "64583fd5c7a53e42cf1d6c29c6823451"
server_key_iv: "5e14c8e99970c518391d964fc24e401d"
server_key_version: "encryptionkey@0"
I20260620 19:08:43.465986 11365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:08:43.476042 11365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:08:43.477798 11365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:08:43.477960 11365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:08:43.478204 11365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:08:43.478508 11365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:08:43.478564 11365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:43.478606 11365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:08:43.478634 11365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:08:43.484733 11365 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:37007
I20260620 19:08:43.484781 11493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:37007 every 8 connection(s)
I20260620 19:08:43.485119 11365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:08:43 dist-test-slave-wd30 krb5kdc[10990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:08:43.491036 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 11365
I20260620 19:08:43.491183 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:08:43.491390 812 external_mini_cluster.cc:1428] Setting key 4e7215ffed8f1468e5374603eca81e7b
I20260620 19:08:43.495609 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:43.485843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:37617 (local address 127.0.203.62:35747)
0620 19:08:43.485927 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:43.485930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:43.486161 (+ 231us) server_negotiation.cc:407] Connection header received
0620 19:08:43.486614 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:43.486615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:43.486626 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:43.486680 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:43.487599 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.488105 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.489102 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.489295 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.489356 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:43.491600 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:43.491601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:43.491601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:43.491607 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:43.494191 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:43.494602 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:43.494603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:43.494604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:43.494660 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:43.494966 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:43.494967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:43.494967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:43.495085 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:43.495234 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:43.495469 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 19:08:43.495547 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:08:43.495795 11496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:43.485785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:35747 (local address 127.0.203.3:37617)
0620 19:08:43.486026 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:43.486061 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:43.486516 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:43.486739 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:43.486743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:43.487325 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:08:43.487508 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:43.487512 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.488212 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.488214 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:43.489005 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:43.489007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.489374 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.489375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:43.489473 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:43.490049 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:08:43.490057 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:08:43.491489 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:08:43.494269 (+ 2780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:43.494275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:43.494277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:43.494507 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:43.494752 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:08:43.494752 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:08:43.494753 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:08:43.494896 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:08:43.495228 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:08:43.495233 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:08:43.495356 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 19:08:43.495441 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 19:08:43.495565 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":95,"threads_started":1}
I20260620 19:08:43.496292 11494 heartbeater.cc:347] Connected to a master server at 127.0.203.62:35747
I20260620 19:08:43.496384 11494 heartbeater.cc:464] Registering TS with master...
I20260620 19:08:43.496518 11494 heartbeater.cc:511] Master 127.0.203.62:35747 requested a full tablet report, sending...
I20260620 19:08:43.496876 11035 ts_manager.cc:194] Registered new tserver with Master: ecdababd7a504f1e98b48aef99a91bd2 (127.0.203.3:37007)
I20260620 19:08:43.497254 11035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:37617
I20260620 19:08:43.503396 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:08:43.510991 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:43.504751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55006 (local address 127.0.203.62:35747)
0620 19:08:43.504804 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:43.504806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:43.504871 (+ 65us) server_negotiation.cc:407] Connection header received
0620 19:08:43.504945 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:43.504946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:43.504962 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:43.505024 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:08:43.505572 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.506048 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.506767 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.506954 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.507022 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:43.508294 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:08:43.508295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:08:43.508295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:08:43.508302 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:08:43.509700 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:43.510060 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:43.510062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:43.510062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:43.510118 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:08:43.510388 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:08:43.510389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:08:43.510389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:08:43.510609 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:08:43.510693 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:43.510797 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 19:08:43.510866 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 19:08:43.512600 11035 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:08:43.514144 11035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:08:43.521596 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:43.516455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:37007 (local address 127.0.0.1:39890)
0620 19:08:43.516638 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:43.516682 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:43.516749 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:43.517265 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:43.517266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:43.517268 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:43.517359 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:43.517384 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.518759 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.518759 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:43.520332 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:43.520336 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.521255 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.521256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:43.521340 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:43.521356 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 19:08:43.521440 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:08:43.521497 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":51,"thread_start_us":106,"threads_started":1}
I20260620 19:08:43.521651 11508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:43.516711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39890 (local address 127.0.203.3:37007)
0620 19:08:43.517081 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:43.517085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:43.517094 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:08:43.517124 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:43.517125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:43.517137 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:43.517464 (+ 327us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:43.517534 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.518650 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.520430 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.521168 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.521237 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:43.521302 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:43.521518 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 19:08:43.521564 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":71,"threads_started":1}
I20260620 19:08:43.522462 11505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:43.516455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:43941 (local address 127.0.0.1:49776)
0620 19:08:43.516649 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:43.516684 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:43.516750 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:43.517260 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:43.517263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:43.517265 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:43.517331 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:43.517333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.518758 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.518759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:43.520349 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:43.520350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.522207 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.522208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:43.522280 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:43.522284 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:08:43.522345 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 19:08:43.522388 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":145,"thread_start_us":49,"threads_started":1}
I20260620 19:08:43.522492 11510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:43.516896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49776 (local address 127.0.203.1:43941)
0620 19:08:43.517081 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:43.517084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:43.517093 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:08:43.517124 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:43.517125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:43.517136 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:43.517238 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:43.517414 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.518632 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.521337 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.522128 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.522196 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:43.522252 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:43.522369 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 19:08:43.522412 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":91,"threads_started":1}
I20260620 19:08:43.522572 11428 tablet_service.cc:1467] Processing CreateTablet for tablet 0d8ed4b71fe34b9fa3b03b94643f9bec (DEFAULT_TABLE table=test-table [id=b9462eec70d1444284c318def6141f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:43.522960 11428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d8ed4b71fe34b9fa3b03b94643f9bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:43.523187 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:43.516463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:44375 (local address 127.0.0.1:59822)
0620 19:08:43.516851 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:43.516860 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:43.516899 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:43.518975 (+ 2076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:43.518976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:43.518979 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:43.519165 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:43.519168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.520105 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.520106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:43.521131 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:43.521134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.522924 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.522925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:43.522988 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:43.522993 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:08:43.523042 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:08:43.523082 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":81,"thread_start_us":132,"threads_started":1}
I20260620 19:08:43.523181 11160 tablet_service.cc:1467] Processing CreateTablet for tablet 0d8ed4b71fe34b9fa3b03b94643f9bec (DEFAULT_TABLE table=test-table [id=b9462eec70d1444284c318def6141f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:43.523504 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:43.516723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.0.203.2:44375)
0620 19:08:43.518818 (+ 2095us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:43.518822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:43.518830 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:08:43.518860 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:43.518861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:43.518873 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:43.518961 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:43.519258 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.519997 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.521251 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.522821 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.522921 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:43.522977 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:43.523190 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 19:08:43.523441 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2063,"thread_start_us":80,"threads_started":1}
I20260620 19:08:43.523517 11160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d8ed4b71fe34b9fa3b03b94643f9bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:43.526520 11511 tablet_bootstrap.cc:492] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2: Bootstrap starting.
I20260620 19:08:43.526901 11512 tablet_bootstrap.cc:492] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa: Bootstrap starting.
I20260620 19:08:43.527372 11511 tablet_bootstrap.cc:654] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:43.527282 11294 tablet_service.cc:1467] Processing CreateTablet for tablet 0d8ed4b71fe34b9fa3b03b94643f9bec (DEFAULT_TABLE table=test-table [id=b9462eec70d1444284c318def6141f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:08:43.527594 11512 tablet_bootstrap.cc:654] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:43.527604 11294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d8ed4b71fe34b9fa3b03b94643f9bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:08:43.527977 11511 log.cc:824] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:43.528173 11512 log.cc:824] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:43.528885 11511 tablet_bootstrap.cc:492] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2: No bootstrap required, opened a new log
I20260620 19:08:43.528901 11512 tablet_bootstrap.cc:492] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa: No bootstrap required, opened a new log
I20260620 19:08:43.528961 11511 ts_tablet_manager.cc:1397] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:08:43.528985 11512 ts_tablet_manager.cc:1397] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:08:43.530341 11512 raft_consensus.cc:348] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdababd7a504f1e98b48aef99a91bd2" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37007 } } peers { permanent_uuid: "ac0cf2efc19c495faa2f1834b957ebaa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43941 } } peers { permanent_uuid: "e4feb4b86005485eba0f8d2dd73b93c8" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44375 } }
I20260620 19:08:43.530515 11512 raft_consensus.cc:374] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:43.530561 11512 raft_consensus.cc:729] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac0cf2efc19c495faa2f1834b957ebaa, State: Initialized, Role: FOLLOWER
I20260620 19:08:43.530699 11512 consensus_queue.cc:260] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdababd7a504f1e98b48aef99a91bd2" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37007 } } peers { permanent_uuid: "ac0cf2efc19c495faa2f1834b957ebaa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43941 } } peers { permanent_uuid: "e4feb4b86005485eba0f8d2dd73b93c8" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44375 } }
I20260620 19:08:43.531018 11512 ts_tablet_manager.cc:1428] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:08:43.531030 11516 tablet_bootstrap.cc:492] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8: Bootstrap starting.
I20260620 19:08:43.530990 11511 raft_consensus.cc:348] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdababd7a504f1e98b48aef99a91bd2" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37007 } } peers { permanent_uuid: "ac0cf2efc19c495faa2f1834b957ebaa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43941 } } peers { permanent_uuid: "e4feb4b86005485eba0f8d2dd73b93c8" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44375 } }
I20260620 19:08:43.531208 11511 raft_consensus.cc:374] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:43.531242 11226 heartbeater.cc:503] Master 127.0.203.62:35747 was elected leader, sending a full tablet report...
I20260620 19:08:43.531260 11511 raft_consensus.cc:729] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecdababd7a504f1e98b48aef99a91bd2, State: Initialized, Role: FOLLOWER
I20260620 19:08:43.531369 11511 consensus_queue.cc:260] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdababd7a504f1e98b48aef99a91bd2" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37007 } } peers { permanent_uuid: "ac0cf2efc19c495faa2f1834b957ebaa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43941 } } peers { permanent_uuid: "e4feb4b86005485eba0f8d2dd73b93c8" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44375 } }
I20260620 19:08:43.531641 11511 ts_tablet_manager.cc:1428] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:08:43.531767 11494 heartbeater.cc:503] Master 127.0.203.62:35747 was elected leader, sending a full tablet report...
I20260620 19:08:43.532114 11516 tablet_bootstrap.cc:654] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8: Neither blocks nor log segments found. Creating new log.
I20260620 19:08:43.532725 11516 log.cc:824] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8: Log is configured to *not* fsync() on all Append() calls
I20260620 19:08:43.533558 11516 tablet_bootstrap.cc:492] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8: No bootstrap required, opened a new log
I20260620 19:08:43.533614 11516 ts_tablet_manager.cc:1397] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 19:08:43.534960 11516 raft_consensus.cc:348] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdababd7a504f1e98b48aef99a91bd2" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37007 } } peers { permanent_uuid: "ac0cf2efc19c495faa2f1834b957ebaa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43941 } } peers { permanent_uuid: "e4feb4b86005485eba0f8d2dd73b93c8" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44375 } }
I20260620 19:08:43.535096 11516 raft_consensus.cc:374] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:08:43.535187 11516 raft_consensus.cc:729] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4feb4b86005485eba0f8d2dd73b93c8, State: Initialized, Role: FOLLOWER
I20260620 19:08:43.535311 11516 consensus_queue.cc:260] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdababd7a504f1e98b48aef99a91bd2" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37007 } } peers { permanent_uuid: "ac0cf2efc19c495faa2f1834b957ebaa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43941 } } peers { permanent_uuid: "e4feb4b86005485eba0f8d2dd73b93c8" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44375 } }
I20260620 19:08:43.535612 11516 ts_tablet_manager.cc:1428] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:08:43.535630 11360 heartbeater.cc:503] Master 127.0.203.62:35747 was elected leader, sending a full tablet report...
W20260620 19:08:43.584771 11361 tablet.cc:2369] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:08:43.667197 11515 raft_consensus.cc:482] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:08:43.667369 11515 raft_consensus.cc:504] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdababd7a504f1e98b48aef99a91bd2" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37007 } } peers { permanent_uuid: "ac0cf2efc19c495faa2f1834b957ebaa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43941 } } peers { permanent_uuid: "e4feb4b86005485eba0f8d2dd73b93c8" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44375 } }
I20260620 19:08:43.667812 11515 leader_election.cc:288] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecdababd7a504f1e98b48aef99a91bd2 (127.0.203.3:37007), e4feb4b86005485eba0f8d2dd73b93c8 (127.0.203.2:44375)
I20260620 19:08:43.670899 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:43.667998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:37007 (local address 127.0.203.1:46219)
0620 19:08:43.668091 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:43.668102 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:43.668153 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:43.668334 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:43.668335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:43.668336 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:43.668419 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:43.668421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.669149 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.669150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:43.670030 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:43.670032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.670611 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.670612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:43.670688 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:43.670691 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:08:43.670762 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:08:43.670806 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 19:08:43.670986 11508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:43.668102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:46219 (local address 127.0.203.3:37007)
0620 19:08:43.668150 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:43.668152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:43.668163 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:08:43.668247 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:43.668249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:43.668261 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:43.668489 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:43.668558 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.669066 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.670122 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.670539 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.670612 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:43.670623 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:43.670852 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 19:08:43.670905 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 19:08:43.671253 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:43.668098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:58923 (local address 127.0.203.2:44375)
0620 19:08:43.668158 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:43.668160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:43.668421 (+ 261us) server_negotiation.cc:407] Connection header received
0620 19:08:43.668526 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:43.668528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:43.668542 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:43.668580 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:08:43.668805 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.669313 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.670344 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.670846 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.670911 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:43.670920 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:43.671109 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 19:08:43.671178 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:08:43.671288 11448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8ed4b71fe34b9fa3b03b94643f9bec" candidate_uuid: "ac0cf2efc19c495faa2f1834b957ebaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecdababd7a504f1e98b48aef99a91bd2" is_pre_election: true
I20260620 19:08:43.671439 11448 raft_consensus.cc:2454] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac0cf2efc19c495faa2f1834b957ebaa in term 0.
I20260620 19:08:43.671526 11520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:43.668003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:44375 (local address 127.0.203.1:58923)
0620 19:08:43.668327 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:43.668337 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:08:43.668429 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:08:43.668660 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:08:43.668660 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:08:43.668662 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:08:43.668750 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:43.668752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.669400 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.669401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:43.670261 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:08:43.670263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.670921 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.670922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:08:43.671009 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:43.671013 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:08:43.671397 (+ 384us) client_negotiation.cc:240] Negotiation successful
0620 19:08:43.671444 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":177,"thread_start_us":106,"threads_started":1}
I20260620 19:08:43.671744 11115 leader_election.cc:302] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac0cf2efc19c495faa2f1834b957ebaa, ecdababd7a504f1e98b48aef99a91bd2; no voters:
I20260620 19:08:43.671861 11314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8ed4b71fe34b9fa3b03b94643f9bec" candidate_uuid: "ac0cf2efc19c495faa2f1834b957ebaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4feb4b86005485eba0f8d2dd73b93c8" is_pre_election: true
I20260620 19:08:43.671898 11515 raft_consensus.cc:2781] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:08:43.671983 11515 raft_consensus.cc:482] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:08:43.672000 11314 raft_consensus.cc:2454] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac0cf2efc19c495faa2f1834b957ebaa in term 0.
I20260620 19:08:43.672017 11515 raft_consensus.cc:3037] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:43.672789 11515 raft_consensus.cc:504] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdababd7a504f1e98b48aef99a91bd2" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37007 } } peers { permanent_uuid: "ac0cf2efc19c495faa2f1834b957ebaa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43941 } } peers { permanent_uuid: "e4feb4b86005485eba0f8d2dd73b93c8" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44375 } }
I20260620 19:08:43.672948 11515 leader_election.cc:288] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [CANDIDATE]: Term 1 election: Requested vote from peers ecdababd7a504f1e98b48aef99a91bd2 (127.0.203.3:37007), e4feb4b86005485eba0f8d2dd73b93c8 (127.0.203.2:44375)
I20260620 19:08:43.673103 11314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8ed4b71fe34b9fa3b03b94643f9bec" candidate_uuid: "ac0cf2efc19c495faa2f1834b957ebaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4feb4b86005485eba0f8d2dd73b93c8"
I20260620 19:08:43.673147 11448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8ed4b71fe34b9fa3b03b94643f9bec" candidate_uuid: "ac0cf2efc19c495faa2f1834b957ebaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecdababd7a504f1e98b48aef99a91bd2"
I20260620 19:08:43.673185 11314 raft_consensus.cc:3037] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:43.673251 11448 raft_consensus.cc:3037] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:08:43.673905 11314 raft_consensus.cc:2454] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac0cf2efc19c495faa2f1834b957ebaa in term 1.
I20260620 19:08:43.673916 11448 raft_consensus.cc:2454] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac0cf2efc19c495faa2f1834b957ebaa in term 1.
I20260620 19:08:43.674072 11115 leader_election.cc:302] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac0cf2efc19c495faa2f1834b957ebaa, ecdababd7a504f1e98b48aef99a91bd2; no voters:
I20260620 19:08:43.674175 11515 raft_consensus.cc:2781] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:08:43.674335 11515 raft_consensus.cc:686] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [term 1 LEADER]: Becoming Leader. State: Replica: ac0cf2efc19c495faa2f1834b957ebaa, State: Running, Role: LEADER
I20260620 19:08:43.674440 11515 consensus_queue.cc:237] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdababd7a504f1e98b48aef99a91bd2" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37007 } } peers { permanent_uuid: "ac0cf2efc19c495faa2f1834b957ebaa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43941 } } peers { permanent_uuid: "e4feb4b86005485eba0f8d2dd73b93c8" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44375 } }
I20260620 19:08:43.675328 11034 catalog_manager.cc:5495] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa reported cstate change: term changed from 0 to 1, leader changed from <none> to ac0cf2efc19c495faa2f1834b957ebaa (127.0.203.1). New cstate: current_term: 1 leader_uuid: "ac0cf2efc19c495faa2f1834b957ebaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdababd7a504f1e98b48aef99a91bd2" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac0cf2efc19c495faa2f1834b957ebaa" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4feb4b86005485eba0f8d2dd73b93c8" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44375 } health_report { overall_health: UNKNOWN } } }
W20260620 19:08:43.679419 11227 tablet.cc:2369] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:08:43.708994 11510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:43.706546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49778 (local address 127.0.203.1:43941)
0620 19:08:43.706641 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 19:08:43.706643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:08:43.706651 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:08:43.706687 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:08:43.706688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:08:43.706708 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:08:43.706768 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:08:43.706874 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.707413 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.708268 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:08:43.708446 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:08:43.708496 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:08:43.708643 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:08:43.708744 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:08:43.708805 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:08:43.708845 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 19:08:43.708862 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 19:08:43.710805 11314 raft_consensus.cc:1261] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8 [term 1 FOLLOWER]: Refusing update from remote peer ac0cf2efc19c495faa2f1834b957ebaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:43.710824 11448 raft_consensus.cc:1261] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2 [term 1 FOLLOWER]: Refusing update from remote peer ac0cf2efc19c495faa2f1834b957ebaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:08:43.711200 11515 consensus_queue.cc:1035] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4feb4b86005485eba0f8d2dd73b93c8" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:43.711364 11521 consensus_queue.cc:1035] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecdababd7a504f1e98b48aef99a91bd2" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:08:43.716527 11034 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:43.716617 11034 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:08:43.717461 11034 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 910275ae92764f408ab27d278f37861a: Sending DeleteTablet for 3 replicas of tablet 0d8ed4b71fe34b9fa3b03b94643f9bec
I20260620 19:08:43.717844 11294 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d8ed4b71fe34b9fa3b03b94643f9bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:43 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:59822
I20260620 19:08:43.717836 11428 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d8ed4b71fe34b9fa3b03b94643f9bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:43 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:39890
I20260620 19:08:43.717864 11160 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d8ed4b71fe34b9fa3b03b94643f9bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:08:43 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:49776
I20260620 19:08:43.718108 11536 tablet_replica.cc:330] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa: stopping tablet replica
I20260620 19:08:43.718210 11536 raft_consensus.cc:2229] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:08:43.718338 11536 raft_consensus.cc:2258] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ac0cf2efc19c495faa2f1834b957ebaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:43.718402 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 11097
I20260620 19:08:43.720197 11535 tablet_replica.cc:330] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8: stopping tablet replica
I20260620 19:08:43.720299 11535 raft_consensus.cc:2229] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:43.720384 11535 raft_consensus.cc:2258] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:43.720564 11534 tablet_replica.cc:330] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2: stopping tablet replica
I20260620 19:08:43.720681 11534 raft_consensus.cc:2229] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:08:43.720791 11534 raft_consensus.cc:2258] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:08:43.720893 11535 ts_tablet_manager.cc:1905] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:43.721202 11534 ts_tablet_manager.cc:1905] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:08:43.722819 11534 ts_tablet_manager.cc:1918] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:43.722923 11534 log.cc:1196] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/0d8ed4b71fe34b9fa3b03b94643f9bec
I20260620 19:08:43.722927 11535 ts_tablet_manager.cc:1918] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:08:43.723006 11535 log.cc:1196] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/0d8ed4b71fe34b9fa3b03b94643f9bec
I20260620 19:08:43.723279 11534 ts_tablet_manager.cc:1939] T 0d8ed4b71fe34b9fa3b03b94643f9bec P ecdababd7a504f1e98b48aef99a91bd2: Deleting consensus metadata
I20260620 19:08:43.723331 11535 ts_tablet_manager.cc:1939] T 0d8ed4b71fe34b9fa3b03b94643f9bec P e4feb4b86005485eba0f8d2dd73b93c8: Deleting consensus metadata
I20260620 19:08:43.723738 11022 catalog_manager.cc:4841] TS e4feb4b86005485eba0f8d2dd73b93c8 (127.0.203.2:44375): tablet 0d8ed4b71fe34b9fa3b03b94643f9bec (table test-table [id=b9462eec70d1444284c318def6141f55]) successfully deleted
I20260620 19:08:43.723874 11021 catalog_manager.cc:4841] TS ecdababd7a504f1e98b48aef99a91bd2 (127.0.203.3:37007): tablet 0d8ed4b71fe34b9fa3b03b94643f9bec (table test-table [id=b9462eec70d1444284c318def6141f55]) successfully deleted
W20260620 19:08:43.724709 11019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.1:43941 (error 108)
I20260620 19:08:43.725226 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:08:43.725092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:43941 (local address 127.0.0.1:49786)
0620 19:08:43.725152 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 19:08:43.725192 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.1:43941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 19:08:43.725242 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 11231
W20260620 19:08:43.725351 11019 catalog_manager.cc:4584] TS ac0cf2efc19c495faa2f1834b957ebaa (127.0.203.1:43941): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d8ed4b71fe34b9fa3b03b94643f9bec: Network error: Client connection negotiation failed: client connection to 127.0.203.1:43941: connect: Connection refused (error 111)
I20260620 19:08:43.730331 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 11365
I20260620 19:08:43.734751 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 11003
2026-06-20T19:08:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2044 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:08:43 dist-test-slave-wd30 krb5kdc[11540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:08:43 dist-test-slave-wd30 krb5kdc[11540](info): set up 2 sockets
Jun 20 19:08:43 dist-test-slave-wd30 krb5kdc[11540](info): commencing operation
krb5kdc: starting...
W20260620 19:08:44.768213 812 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:08:44 dist-test-slave-wd30 krb5kdc[11540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:08:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:08:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.203.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.203.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.203.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/rangeradmin_127.0.203.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.203.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/rangeradmin_127.0.203.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.203.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.203.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.203.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/rangerlookup_127.0.203.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.203.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/rangerlookup_127.0.203.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.203.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.203.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.203.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/HTTP_127.0.203.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.203.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/HTTP_127.0.203.20@KRBTEST.COM.keytab.
I20260620 19:08:44.931908 812 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskeuOXoM/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/postgres -l logfile start
2026-06-20 19:08:47.403 UTC [11570] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 19:08:47.404 UTC [11570] LOG: listening on IPv4 address "127.0.203.20", port 47227
2026-06-20 19:08:47.407 UTC [11570] LOG: listening on Unix socket "/tmp/.s.PGSQL.47227"
2026-06-20 19:08:47.440 UTC [11573] LOG: database system was shut down at 2026-06-20 19:08:45 UTC
2026-06-20 19:08:47.444 UTC [11570] LOG: database system is ready to accept connections
I20260620 19:08:48.396073 812 mini_postgres.cc:96] Postgres bound to 47227
2026-06-20 19:08:48.402 UTC [11581] FATAL: database "slave" does not exist
127.0.203.20:47227 - accepting connections
I20260620 19:08:48.402626 812 mini_ranger.cc:161] Starting Ranger...
I20260620 19:08:48.417357 812 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 19:08:48.876689 812 mini_ranger.cc:87] Created ranger Postgres database
I20260620 19:08:48.876781 812 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ranger-admin
2026-06-20 19:08:49,166 [I] DB FLAVOR :POSTGRES
2026-06-20 19:08:49,166 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:08:49,166 [I] Checking connection..
2026-06-20 19:08:49,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 19:08:49,412 [I] Checking connection passed.
2026-06-20 19:08:49,412 [I] --------- Verifying version history table ---------
2026-06-20 19:08:49,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:08:49,645 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:08:49,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:08:49,888 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:08:49,888 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 19:08:49,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 19:08:50.147 UTC [11645] WARNING: there is no transaction in progress
2026-06-20 19:08:50,153 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 19:08:50,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:08:50,418 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 19:08:50,418 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:08:50,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 19:08:50,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 19:08:50,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wd30.c.gcp-upstream.internal','N') ;"
2026-06-20 19:08:51,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:08:51,413 [I] Table x_portal_user does not exist in database ranger
2026-06-20 19:08:51,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:08:51,681 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 19:08:51,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:08:51,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:08:52,193 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 19:08:52,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 19:08:53.168 UTC [11753] WARNING: there is no transaction in progress
2026-06-20 19:08:53.184 UTC [11753] WARNING: there is no transaction in progress
2026-06-20 19:08:53.200 UTC [11753] WARNING: there is no transaction in progress
2026-06-20 19:08:53.215 UTC [11753] WARNING: there is no transaction in progress
2026-06-20 19:08:53.231 UTC [11753] WARNING: there is no transaction in progress
2026-06-20 19:08:53.251 UTC [11753] WARNING: there is no transaction in progress
2026-06-20 19:08:53.316 UTC [11753] WARNING: there is no transaction in progress
2026-06-20 19:08:53.323 UTC [11753] WARNING: there is no transaction in progress
2026-06-20 19:08:53.338 UTC [11753] WARNING: there is no transaction in progress
2026-06-20 19:08:53.348 UTC [11753] WARNING: there is no transaction in progress
2026-06-20 19:08:53.359 UTC [11753] WARNING: there is no transaction in progress
2026-06-20 19:08:53.373 UTC [11753] WARNING: there is no transaction in progress
2026-06-20 19:08:53.383 UTC [11753] WARNING: there is no transaction in progress
2026-06-20 19:08:53.390 UTC [11753] WARNING: there is no transaction in progress
2026-06-20 19:08:53.396 UTC [11753] WARNING: there is no transaction in progress
2026-06-20 19:08:53.403 UTC [11753] WARNING: there is no transaction in progress
2026-06-20 19:08:54,058 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 19:08:54,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 19:08:54,293 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 19:08:54,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:08:54,556 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 19:08:54,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:08:54,832 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 19:08:54,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:08:55,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:08:55,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wd30.c.gcp-upstream.internal';"
2026-06-20 19:08:55,602 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 19:08:55,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 19:08:55,839 [I] DB_PATCHES have already been applied
I20260620 19:08:55.845336 812 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ranger-admin:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskeuOXoM/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 19:08:55.845408 812 mini_ranger.cc:193] Using host: 127.0.203.20
I20260620 19:08:55.848470 812 mini_ranger.cc:227] Ranger admin URL: http://127.0.203.20:47057
Jun 20, 2026 7:08:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:08:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:08:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 7:08:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 7:08:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 7:08:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 7:08:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 19:08:56.962589 812 mini_ranger.cc:160] Time spent starting Ranger: real 8.560s user 0.000s sys 0.009s
Jun 20, 2026 7:09:17 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:09:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Jun 20, 2026 7:09:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:09:17 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:09:17 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:09:18 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 19:09:19.257432 812 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.203.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.203.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.203.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/rangerkms_127.0.203.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.203.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/rangerkms_127.0.203.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.203.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.203.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.203.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/HTTP_127.0.203.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.203.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/HTTP_127.0.203.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 19:09:19 dist-test-slave-wd30 krb5kdc[11540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982559, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 19:09:19.336536 812 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 19:09:19.345953 812 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 19:09:20.168751 812 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 19:09:20.168838 812 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ranger-kms
I20260620 19:09:20.168859 812 mini_ranger_kms.cc:227] Using postgres at 127.0.203.20:47227
2026-06-20 19:09:20,383 [I] DB FLAVOR :POSTGRES
2026-06-20 19:09:20,383 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:09:20,383 [I] Checking connection
2026-06-20 19:09:20,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 19:09:20,659 [I] connection success
2026-06-20 19:09:20,660 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 19:09:20,660 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 19:09:20,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 19:09:20,923 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 19:09:20,923 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:09:20,924 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 19:09:20,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:47227/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 19:09:21,217 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 19:09:21.739435 812 mini_ranger_kms.cc:310] Created kms service
I20260620 19:09:21.942800 812 mini_ranger_kms.cc:326] Created kudu user
I20260620 19:09:21.988871 812 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 19:09:21 dist-test-slave-wd30 krb5kdc[11540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982559, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
I20260620 19:09:22.382455 812 mini_ranger_kms.cc:384] Added ranger policy
I20260620 19:09:22.382575 812 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ranger-kms:/tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 19:09:22.382596 812 mini_ranger_kms.cc:240] Using host: 127.0.203.20
I20260620 19:09:22.386013 812 mini_ranger_kms.cc:276] Ranger KMS PID: 11933
I20260620 19:09:22.386152 812 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.203.20:48959
Jun 20, 2026 7:09:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:09:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:09:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:09:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:09:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:09:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 19:09:24.643190 812 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.307s user 0.000s sys 0.197s
I20260620 19:09:24.643303 812 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 19:09:24.643397 812 mini_ranger_kms.cc:401] 127.0.203.20:48959/kms/v1/keys
Jun 20 19:09:24 dist-test-slave-wd30 krb5kdc[11540](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982564, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.203.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 19:09:25 dist-test-slave-wd30 krb5kdc[11540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982564, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.203.20@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
Jun 20 19:09:25 dist-test-slave-wd30 krb5kdc[11540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982564, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.203.20@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
Jun 20 19:09:27 dist-test-slave-wd30 krb5kdc[11540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:09:27.931744 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:39813
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:39813
--ranger_config_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.20:48959/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:28.034516 11984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:28.034767 11984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:28.034802 11984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:28.036046 11984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:28.036113 11984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:28.036135 11984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:28.036155 11984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:28.036187 11984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:28.037743 11984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.203.20:48959/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:39813
--ranger_config_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:39813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:09:28.038022 11984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:28.038245 11984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:28.043689 11989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:28.043781 11984 server_base.cc:1034] running on GCE node
W20260620 19:09:28.043900 11992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:28.043905 11990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:28.044193 11984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:28.044508 11984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:28.045688 11984 hybrid_clock.cc:648] HybridClock initialized: now 1781982568045675 us; error 32 us; skew 500 ppm
Jun 20 19:09:28 dist-test-slave-wd30 krb5kdc[11540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:28.058326 11984 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:09:28.058928 11984 webserver.cc:460] Webserver started at http://127.0.203.62:46553/ using document root <none> and password file <none>
I20260620 19:09:28.059227 11984 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:28.059293 11984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:28.059446 11984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:09:28 dist-test-slave-wd30 krb5kdc[11540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
I20260620 19:09:28.121117 11984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "e7dbb5f1c7324792a928e8300de4cb69"
format_stamp: "Formatted at 2026-06-20 19:09:28 on dist-test-slave-wd30"
server_key: "4c7963186127382e9c25dab5cc8cfe4d"
server_key_iv: "4a9f09160f9b2030b880f1c64d959cdb"
server_key_version: "kuduclusterkey@0"
I20260620 19:09:28.121539 11984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "e7dbb5f1c7324792a928e8300de4cb69"
format_stamp: "Formatted at 2026-06-20 19:09:28 on dist-test-slave-wd30"
server_key: "4c7963186127382e9c25dab5cc8cfe4d"
server_key_iv: "4a9f09160f9b2030b880f1c64d959cdb"
server_key_version: "kuduclusterkey@0"
I20260620 19:09:28.123057 11984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260620 19:09:28.138087 11999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:28.138391 11984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:09:28.138509 11984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "e7dbb5f1c7324792a928e8300de4cb69"
format_stamp: "Formatted at 2026-06-20 19:09:28 on dist-test-slave-wd30"
server_key: "4c7963186127382e9c25dab5cc8cfe4d"
server_key_iv: "4a9f09160f9b2030b880f1c64d959cdb"
server_key_version: "kuduclusterkey@0"
I20260620 19:09:28.138615 11984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:28.153185 11984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:28.155450 11984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:28.155598 11984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:28.160574 11984 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:39813
I20260620 19:09:28.160641 12051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:39813 every 8 connection(s)
I20260620 19:09:28.161011 11984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:28.162159 12052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:28.164650 12052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69: Bootstrap starting.
I20260620 19:09:28.165257 12052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:28.165562 12052 log.cc:824] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:28.166293 12052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69: No bootstrap required, opened a new log
I20260620 19:09:28.168553 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 11984
I20260620 19:09:28.168660 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:09:28.168632 12052 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7dbb5f1c7324792a928e8300de4cb69" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39813 } }
I20260620 19:09:28.168843 812 external_mini_cluster.cc:1428] Setting key 665349324b0d1204b60ff09fe6a6d467
I20260620 19:09:28.168887 12052 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:28.168959 12052 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7dbb5f1c7324792a928e8300de4cb69, State: Initialized, Role: FOLLOWER
I20260620 19:09:28.169139 12052 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7dbb5f1c7324792a928e8300de4cb69" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39813 } }
I20260620 19:09:28.169265 12052 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:28.169322 12052 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:28.169380 12052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:28.170169 12052 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7dbb5f1c7324792a928e8300de4cb69" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39813 } }
I20260620 19:09:28.170284 12052 leader_election.cc:302] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7dbb5f1c7324792a928e8300de4cb69; no voters:
I20260620 19:09:28.170450 12052 leader_election.cc:288] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:28.170496 12057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:28.170668 12057 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69 [term 1 LEADER]: Becoming Leader. State: Replica: e7dbb5f1c7324792a928e8300de4cb69, State: Running, Role: LEADER
I20260620 19:09:28.170723 12052 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:28.170787 12057 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7dbb5f1c7324792a928e8300de4cb69" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39813 } }
I20260620 19:09:28.171607 12057 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7dbb5f1c7324792a928e8300de4cb69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7dbb5f1c7324792a928e8300de4cb69" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39813 } } }
I20260620 19:09:28.171749 12057 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:28.171990 12068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:28.172364 12052 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 19:09:28.172672 12058 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7dbb5f1c7324792a928e8300de4cb69. Latest consensus state: current_term: 1 leader_uuid: "e7dbb5f1c7324792a928e8300de4cb69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7dbb5f1c7324792a928e8300de4cb69" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39813 } } }
I20260620 19:09:28.172780 12058 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:28.176443 12068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 19:09:28 dist-test-slave-wd30 krb5kdc[11540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:09:28.186383 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:28.169492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50970 (local address 127.0.203.62:39813)
0620 19:09:28.169683 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:28.169693 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:28.171112 (+ 1419us) server_negotiation.cc:407] Connection header received
0620 19:09:28.171297 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:28.171305 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:28.171362 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:28.171522 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:28.172378 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:28.176153 (+ 3775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:28.177169 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:28.178156 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:28.178312 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:28.181550 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:28.181555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:28.181555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:28.181571 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:28.184601 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:28.184987 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:28.184991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:28.184991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:28.185104 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:28.185336 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:28.185337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:28.185337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:28.185669 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:28.185840 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:28.185956 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 19:09:28.186088 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
I20260620 19:09:28.188755 12068 catalog_manager.cc:1347] Generated new cluster ID: 761ce61d65fe4f2cbae5bb80266a602e
I20260620 19:09:28.188818 12068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:28.206194 12068 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:28.207002 12068 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:28.216272 12068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69: Generated new TSK 0
I20260620 19:09:28.216615 12068 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
I20260620 19:09:28.982668 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.20:48959/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:39813
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:29.085419 12097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:29.085618 12097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:29.085641 12097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:29.086796 12097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:29.086836 12097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:29.086862 12097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:09:29.088423 12097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.203.20:48959/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:39813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:09:29.088685 12097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:29.088925 12097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:29.093956 12103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:29.093969 12102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:29.094230 12097 server_base.cc:1034] running on GCE node
W20260620 19:09:29.094126 12105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:29.094552 12097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:29.094843 12097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:29.096014 12097 hybrid_clock.cc:648] HybridClock initialized: now 1781982569096007 us; error 35 us; skew 500 ppm
Jun 20 19:09:29 dist-test-slave-wd30 krb5kdc[11540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:29.099463 12097 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:09:29.099963 12097 webserver.cc:460] Webserver started at http://127.0.203.1:40163/ using document root <none> and password file <none>
I20260620 19:09:29.100194 12097 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:29.100237 12097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:29.100370 12097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:09:29 dist-test-slave-wd30 krb5kdc[11540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
I20260620 19:09:29.114522 12097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "ecd1a43fef7d4ba88af942fb70e645a7"
format_stamp: "Formatted at 2026-06-20 19:09:29 on dist-test-slave-wd30"
server_key: "ab713bcfb756df332576db72845068af"
server_key_iv: "9f42edf9cb1f728b1f9f13d76cb31f07"
server_key_version: "kuduclusterkey@0"
I20260620 19:09:29.114996 12097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "ecd1a43fef7d4ba88af942fb70e645a7"
format_stamp: "Formatted at 2026-06-20 19:09:29 on dist-test-slave-wd30"
server_key: "ab713bcfb756df332576db72845068af"
server_key_iv: "9f42edf9cb1f728b1f9f13d76cb31f07"
server_key_version: "kuduclusterkey@0"
I20260620 19:09:29.120501 12097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.001s
I20260620 19:09:29.134860 12114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:29.135109 12097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:09:29.135239 12097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "ecd1a43fef7d4ba88af942fb70e645a7"
format_stamp: "Formatted at 2026-06-20 19:09:29 on dist-test-slave-wd30"
server_key: "ab713bcfb756df332576db72845068af"
server_key_iv: "9f42edf9cb1f728b1f9f13d76cb31f07"
server_key_version: "kuduclusterkey@0"
I20260620 19:09:29.135339 12097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:29.157553 12097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:29.160346 12097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:29.160627 12097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:29.160934 12097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:29.161370 12097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:29.161465 12097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:29.161576 12097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:29.161638 12097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:29.168674 12097 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:44791
I20260620 19:09:29.169088 12097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:09:29.169450 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 12097
I20260620 19:09:29.169551 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:09:29.169800 812 external_mini_cluster.cc:1428] Setting key 815b11e59d7cf5190f5cf158ae7a4285
I20260620 19:09:29.171403 12227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:44791 every 8 connection(s)
Jun 20 19:09:29 dist-test-slave-wd30 krb5kdc[11540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.203.2@KRBTEST.COM; defaulting to no policy
I20260620 19:09:29.182247 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:29.169954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:39813 (local address 127.0.203.1:48291)
0620 19:09:29.171571 (+ 1617us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:29.171612 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:29.172311 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:29.172732 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:29.172737 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:29.172972 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:29.173265 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:29.173272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.174302 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.174303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:29.175584 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:29.175587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.176108 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.176109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:29.176261 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:29.176892 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:29.176902 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:29.178501 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:29.180522 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:29.180530 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:29.180533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:29.180857 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:29.181094 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:29.181095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:29.181095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:29.181243 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:29.181561 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:29.181567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:29.181739 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 19:09:29.181851 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 19:09:29.181999 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1571,"thread_start_us":82,"threads_started":1}
I20260620 19:09:29.182544 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:29.170232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:48291 (local address 127.0.203.62:39813)
0620 19:09:29.172494 (+ 2262us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:29.172496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:29.172504 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:09:29.172527 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:29.172527 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:29.172540 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:29.172600 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:29.173378 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.174192 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.175683 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.175981 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.176044 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:29.178606 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:29.178607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:29.178608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:29.178615 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:29.180422 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:29.180956 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:29.180958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:29.180958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:29.181019 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:29.181327 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:29.181328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:29.181328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:29.181477 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:29.181553 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:29.182371 (+ 818us) server_negotiation.cc:299] Negotiation successful
0620 19:09:29.182450 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2233,"thread_start_us":75,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.2@KRBTEST.COM" created.
I20260620 19:09:29.183286 12228 heartbeater.cc:347] Connected to a master server at 127.0.203.62:39813
I20260620 19:09:29.183390 12228 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:29.183602 12228 heartbeater.cc:511] Master 127.0.203.62:39813 requested a full tablet report, sending...
I20260620 19:09:29.184211 12016 ts_manager.cc:194] Registered new tserver with Master: ecd1a43fef7d4ba88af942fb70e645a7 (127.0.203.1:44791)
I20260620 19:09:29.184993 12016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:48291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
I20260620 19:09:29.202737 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.20:48959/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:39813
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:29.304929 12234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:29.305171 12234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:29.305215 12234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:29.306430 12234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:29.306507 12234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:29.306557 12234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:09:29.308162 12234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.203.20:48959/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:39813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:09:29.308451 12234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:29.308715 12234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:29.314424 12244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:29.314797 12242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:29.314433 12241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:29.315179 12234 server_base.cc:1034] running on GCE node
I20260620 19:09:29.315402 12234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:29.315694 12234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:29.316883 12234 hybrid_clock.cc:648] HybridClock initialized: now 1781982569316862 us; error 38 us; skew 500 ppm
Jun 20 19:09:29 dist-test-slave-wd30 krb5kdc[11540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:29.320317 12234 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:09:29.320883 12234 webserver.cc:460] Webserver started at http://127.0.203.2:40041/ using document root <none> and password file <none>
I20260620 19:09:29.321204 12234 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:29.321278 12234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:29.321477 12234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:09:29 dist-test-slave-wd30 krb5kdc[11540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
Jun 20 19:09:29 dist-test-slave-wd30 krb5kdc[11540](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:29.333612 12234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "13989430db72432cba39c31e17d10ad9"
format_stamp: "Formatted at 2026-06-20 19:09:29 on dist-test-slave-wd30"
server_key: "3385737dd4b866d6cc4466b7e2c63380"
server_key_iv: "44e5872ff5504e2fe5d43f37185a1fe7"
server_key_version: "kuduclusterkey@0"
I20260620 19:09:29.334333 12234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "13989430db72432cba39c31e17d10ad9"
format_stamp: "Formatted at 2026-06-20 19:09:29 on dist-test-slave-wd30"
server_key: "3385737dd4b866d6cc4466b7e2c63380"
server_key_iv: "44e5872ff5504e2fe5d43f37185a1fe7"
server_key_version: "kuduclusterkey@0"
I20260620 19:09:29.339630 12234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.002s
I20260620 19:09:29.361954 12251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:29.362222 12234 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:09:29.362315 12234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "13989430db72432cba39c31e17d10ad9"
format_stamp: "Formatted at 2026-06-20 19:09:29 on dist-test-slave-wd30"
server_key: "3385737dd4b866d6cc4466b7e2c63380"
server_key_iv: "44e5872ff5504e2fe5d43f37185a1fe7"
server_key_version: "kuduclusterkey@0"
I20260620 19:09:29.362412 12234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:29.378631 12234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:29.383809 12234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:29.384037 12234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:29.384317 12234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:29.384706 12234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:29.384780 12234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:29.384831 12234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:29.384865 12234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:29.392221 12234 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:36811
I20260620 19:09:29.392279 12364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:36811 every 8 connection(s)
I20260620 19:09:29.392645 12234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:09:29 dist-test-slave-wd30 krb5kdc[11540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:09:29.401880 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:29.393420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:43297 (local address 127.0.203.62:39813)
0620 19:09:29.393474 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:29.393477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:29.393692 (+ 215us) server_negotiation.cc:407] Connection header received
0620 19:09:29.394210 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:29.394212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:29.394224 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:29.394285 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:29.394837 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.395415 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.396460 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.396664 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.396752 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:29.398784 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:29.398786 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:29.398787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:29.398795 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:29.400428 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:29.400849 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:29.400850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:29.400850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:29.400910 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:29.401170 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:29.401171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:29.401171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:29.401291 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:29.401370 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:29.401698 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 19:09:29.401788 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 19:09:29.402107 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 12234
I20260620 19:09:29.402208 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:09:29.402390 812 external_mini_cluster.cc:1428] Setting key 19af5957fe924cfce66e4c9dc8ec19aa
I20260620 19:09:29.402796 12367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:29.393418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:39813 (local address 127.0.203.2:43297)
0620 19:09:29.393576 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:29.393618 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:29.394097 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:29.394364 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:29.394368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:29.394546 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:29.394737 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:29.394741 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.395507 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.395509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:29.396358 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:29.396360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.396777 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.396778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:29.396900 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:29.397430 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:29.397438 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:29.398689 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:29.400529 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:29.400535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:29.400537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:29.400768 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:29.400980 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:29.400981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:29.400982 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:29.401099 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:29.401361 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:29.401367 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:29.401523 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 19:09:29.401676 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 19:09:29.401824 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260620 19:09:29.404758 12365 heartbeater.cc:347] Connected to a master server at 127.0.203.62:39813
I20260620 19:09:29.404903 12365 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:29.405131 12365 heartbeater.cc:511] Master 127.0.203.62:39813 requested a full tablet report, sending...
I20260620 19:09:29.405586 12016 ts_manager.cc:194] Registered new tserver with Master: 13989430db72432cba39c31e17d10ad9 (127.0.203.2:36811)
I20260620 19:09:29.405962 12016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:43297
WARNING: no policy specified for kudu/127.0.203.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
I20260620 19:09:29.429939 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.20:48959/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:39813
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:29.519761 12371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:29.520110 12371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:29.520216 12371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:29.521950 12371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:29.522090 12371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:29.522176 12371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:09:29.523813 12371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.203.20:48959/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:39813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:09:29.524086 12371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:29.524392 12371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:29.529398 12376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:29.529398 12379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:29.529582 12377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:29.530325 12371 server_base.cc:1034] running on GCE node
I20260620 19:09:29.530552 12371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:29.530798 12371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:29.531973 12371 hybrid_clock.cc:648] HybridClock initialized: now 1781982569531934 us; error 54 us; skew 500 ppm
Jun 20 19:09:29 dist-test-slave-wd30 krb5kdc[11540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:29.535001 12371 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:09:29.535494 12371 webserver.cc:460] Webserver started at http://127.0.203.3:43863/ using document root <none> and password file <none>
I20260620 19:09:29.535746 12371 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:29.535818 12371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:29.535964 12371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:09:29 dist-test-slave-wd30 krb5kdc[11540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
I20260620 19:09:29.556370 12371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "0c4163f3f235491c8dd83de7f9941689"
format_stamp: "Formatted at 2026-06-20 19:09:29 on dist-test-slave-wd30"
server_key: "879e3783d1d7f5834f8403abb15de11a"
server_key_iv: "06a712da4feb8e1b70c3a3e6adf3e426"
server_key_version: "kuduclusterkey@0"
I20260620 19:09:29.556865 12371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "0c4163f3f235491c8dd83de7f9941689"
format_stamp: "Formatted at 2026-06-20 19:09:29 on dist-test-slave-wd30"
server_key: "879e3783d1d7f5834f8403abb15de11a"
server_key_iv: "06a712da4feb8e1b70c3a3e6adf3e426"
server_key_version: "kuduclusterkey@0"
I20260620 19:09:29.559743 12371 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:09:29.579360 12386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:29.579612 12371 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260620 19:09:29.579705 12371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "0c4163f3f235491c8dd83de7f9941689"
format_stamp: "Formatted at 2026-06-20 19:09:29 on dist-test-slave-wd30"
server_key: "879e3783d1d7f5834f8403abb15de11a"
server_key_iv: "06a712da4feb8e1b70c3a3e6adf3e426"
server_key_version: "kuduclusterkey@0"
I20260620 19:09:29.579807 12371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:29.592547 12371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:29.595069 12371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:29.595325 12371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:29.595587 12371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:29.596074 12371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:29.596169 12371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:29.596243 12371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:29.596297 12371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:29.605448 12371 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:46251
I20260620 19:09:29.606032 12371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:09:29.606617 12499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:46251 every 8 connection(s)
Jun 20 19:09:29 dist-test-slave-wd30 krb5kdc[11540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:09:29.616137 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 12371
I20260620 19:09:29.616252 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:09:29.616444 812 external_mini_cluster.cc:1428] Setting key adb41da9fbfddfa965ae29819b77cb30
I20260620 19:09:29.620489 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:29.607649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:47797 (local address 127.0.203.62:39813)
0620 19:09:29.607695 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:29.607697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:29.608226 (+ 529us) server_negotiation.cc:407] Connection header received
0620 19:09:29.608946 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:29.608948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:29.608960 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:29.608991 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:29.609841 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.610670 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.612289 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.612600 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.612644 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:29.615406 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:29.615408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:29.615408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:29.615420 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:29.618723 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:29.619337 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:29.619338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:29.619338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:29.619400 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:29.619692 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:29.619693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:29.619693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:29.619831 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:29.619884 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:29.620300 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 19:09:29.620388 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:09:29.621263 12502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:29.607307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:39813 (local address 127.0.203.3:47797)
0620 19:09:29.608074 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:29.608116 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:29.608818 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:29.609192 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:29.609197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:29.609441 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:29.609725 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:29.609732 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.610793 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.610794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:29.612180 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:29.612183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.612746 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.612747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:29.612892 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:29.613549 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:29.613560 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:29.615294 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:29.618857 (+ 3563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:29.618865 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:29.618867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:29.619243 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:29.619471 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:29.619472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:29.619472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:29.619612 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:29.619917 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:29.619923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:29.620111 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 19:09:29.620839 (+ 728us) client_negotiation.cc:240] Negotiation successful
0620 19:09:29.621012 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":85,"threads_started":1}
I20260620 19:09:29.621940 12500 heartbeater.cc:347] Connected to a master server at 127.0.203.62:39813
I20260620 19:09:29.622048 12500 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:29.622254 12500 heartbeater.cc:511] Master 127.0.203.62:39813 requested a full tablet report, sending...
I20260620 19:09:29.622674 12016 ts_manager.cc:194] Registered new tserver with Master: 0c4163f3f235491c8dd83de7f9941689 (127.0.203.3:46251)
I20260620 19:09:29.623260 12016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:47797
I20260620 19:09:29.629494 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:09:29.641089 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:29.631239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.0.203.62:39813)
0620 19:09:29.631280 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:29.631282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:29.633465 (+ 2183us) server_negotiation.cc:407] Connection header received
0620 19:09:29.633562 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:29.633563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:29.633574 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:29.633603 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:29.634384 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.635200 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.636299 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.636598 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.636638 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:29.638210 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:29.638211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:29.638212 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:29.638220 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:29.639914 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:29.640302 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:29.640303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:29.640304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:29.640356 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:29.640542 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:29.640542 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:29.640543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:29.640657 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:29.640727 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:29.640923 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 19:09:29.640999 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:09:29.643699 12016 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:09:29.646162 12016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:09:29.653662 12512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:29.648847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:44791 (local address 127.0.0.1:39920)
0620 19:09:29.649236 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:29.649277 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:29.649402 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:29.649760 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:29.649763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:29.649765 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:29.649835 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:29.649838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.651066 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.651067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:29.652326 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:29.652329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.653353 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.653354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:29.653444 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:29.653458 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 19:09:29.653525 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:09:29.653578 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":67,"threads_started":1}
I20260620 19:09:29.654402 12513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:29.649094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39920 (local address 127.0.203.1:44791)
0620 19:09:29.649535 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:29.649540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:29.649550 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:09:29.649592 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:29.649594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:29.649609 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:29.649673 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:29.649922 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.650962 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.652443 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.653171 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.653243 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:29.653316 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:29.653871 (+ 555us) server_negotiation.cc:299] Negotiation successful
0620 19:09:29.653917 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":48,"threads_started":1}
I20260620 19:09:29.655100 12153 tablet_service.cc:1467] Processing CreateTablet for tablet f3cbf5feb8fd42ee8a82a66695b057f1 (DEFAULT_TABLE table=test-table [id=dc7625d4827c4c21a08927f64bc127a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:29.655449 12514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:29.649488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:46251 (local address 127.0.0.1:36460)
0620 19:09:29.650208 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:29.650215 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:29.650243 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:29.650507 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:29.650508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:29.650510 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:29.650564 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:29.650566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.652181 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.652181 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:29.653342 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:29.653344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.655138 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.655139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:29.655226 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:29.655231 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:09:29.655266 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 19:09:29.655369 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":58,"threads_started":1}
I20260620 19:09:29.655519 12153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3cbf5feb8fd42ee8a82a66695b057f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:29.656078 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:29.649832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36460 (local address 127.0.203.3:46251)
0620 19:09:29.650360 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:29.650364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:29.650373 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:09:29.650407 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:29.650409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:29.650424 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:29.650629 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:29.650697 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.652084 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.653463 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.655018 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.655839 (+ 821us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:29.655931 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:29.655956 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 19:09:29.655999 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":97,"threads_started":1}
I20260620 19:09:29.656867 12423 tablet_service.cc:1467] Processing CreateTablet for tablet f3cbf5feb8fd42ee8a82a66695b057f1 (DEFAULT_TABLE table=test-table [id=dc7625d4827c4c21a08927f64bc127a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:29.657241 12423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3cbf5feb8fd42ee8a82a66695b057f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:29.659359 12518 tablet_bootstrap.cc:492] T f3cbf5feb8fd42ee8a82a66695b057f1 P ecd1a43fef7d4ba88af942fb70e645a7: Bootstrap starting.
I20260620 19:09:29.660372 12518 tablet_bootstrap.cc:654] T f3cbf5feb8fd42ee8a82a66695b057f1 P ecd1a43fef7d4ba88af942fb70e645a7: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:29.660717 12518 log.cc:824] T f3cbf5feb8fd42ee8a82a66695b057f1 P ecd1a43fef7d4ba88af942fb70e645a7: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:29.661258 12515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:29.648858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:36811 (local address 127.0.0.1:40304)
0620 19:09:29.650083 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:29.650092 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:29.650129 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:29.656602 (+ 6473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:29.656603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:29.656606 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:29.656643 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:29.656645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.658377 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.658378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:29.659558 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:29.659561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.661030 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.661031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:29.661097 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:29.661102 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:09:29.661130 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 19:09:29.661183 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1208,"thread_start_us":529,"threads_started":1}
I20260620 19:09:29.661521 12518 tablet_bootstrap.cc:492] T f3cbf5feb8fd42ee8a82a66695b057f1 P ecd1a43fef7d4ba88af942fb70e645a7: No bootstrap required, opened a new log
I20260620 19:09:29.661594 12518 ts_tablet_manager.cc:1397] T f3cbf5feb8fd42ee8a82a66695b057f1 P ecd1a43fef7d4ba88af942fb70e645a7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:09:29.661886 12516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:29.649693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40304 (local address 127.0.203.2:36811)
0620 19:09:29.656411 (+ 6718us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:29.656415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:29.656427 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:09:29.656475 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:29.656477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:29.656494 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:29.656714 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:29.657106 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.658275 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.659722 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.660934 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.661537 (+ 603us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:29.661740 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:29.661767 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 19:09:29.661808 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6674,"thread_start_us":86,"threads_started":1}
I20260620 19:09:29.662343 12519 tablet_bootstrap.cc:492] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689: Bootstrap starting.
I20260620 19:09:29.662776 12299 tablet_service.cc:1467] Processing CreateTablet for tablet f3cbf5feb8fd42ee8a82a66695b057f1 (DEFAULT_TABLE table=test-table [id=dc7625d4827c4c21a08927f64bc127a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:29.663197 12299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3cbf5feb8fd42ee8a82a66695b057f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:29.663504 12519 tablet_bootstrap.cc:654] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:29.663842 12519 log.cc:824] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:29.663960 12518 raft_consensus.cc:348] T f3cbf5feb8fd42ee8a82a66695b057f1 P ecd1a43fef7d4ba88af942fb70e645a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13989430db72432cba39c31e17d10ad9" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36811 } } peers { permanent_uuid: "0c4163f3f235491c8dd83de7f9941689" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 46251 } } peers { permanent_uuid: "ecd1a43fef7d4ba88af942fb70e645a7" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 44791 } }
I20260620 19:09:29.664160 12518 raft_consensus.cc:374] T f3cbf5feb8fd42ee8a82a66695b057f1 P ecd1a43fef7d4ba88af942fb70e645a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:29.664201 12518 raft_consensus.cc:729] T f3cbf5feb8fd42ee8a82a66695b057f1 P ecd1a43fef7d4ba88af942fb70e645a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecd1a43fef7d4ba88af942fb70e645a7, State: Initialized, Role: FOLLOWER
I20260620 19:09:29.664330 12518 consensus_queue.cc:260] T f3cbf5feb8fd42ee8a82a66695b057f1 P ecd1a43fef7d4ba88af942fb70e645a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13989430db72432cba39c31e17d10ad9" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36811 } } peers { permanent_uuid: "0c4163f3f235491c8dd83de7f9941689" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 46251 } } peers { permanent_uuid: "ecd1a43fef7d4ba88af942fb70e645a7" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 44791 } }
I20260620 19:09:29.664577 12518 ts_tablet_manager.cc:1428] T f3cbf5feb8fd42ee8a82a66695b057f1 P ecd1a43fef7d4ba88af942fb70e645a7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:29.664690 12228 heartbeater.cc:503] Master 127.0.203.62:39813 was elected leader, sending a full tablet report...
I20260620 19:09:29.666496 12519 tablet_bootstrap.cc:492] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689: No bootstrap required, opened a new log
I20260620 19:09:29.666572 12519 ts_tablet_manager.cc:1397] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 19:09:29.668969 12519 raft_consensus.cc:348] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13989430db72432cba39c31e17d10ad9" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36811 } } peers { permanent_uuid: "0c4163f3f235491c8dd83de7f9941689" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 46251 } } peers { permanent_uuid: "ecd1a43fef7d4ba88af942fb70e645a7" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 44791 } }
I20260620 19:09:29.669155 12519 raft_consensus.cc:374] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:29.669188 12519 raft_consensus.cc:729] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4163f3f235491c8dd83de7f9941689, State: Initialized, Role: FOLLOWER
I20260620 19:09:29.669327 12519 consensus_queue.cc:260] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13989430db72432cba39c31e17d10ad9" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36811 } } peers { permanent_uuid: "0c4163f3f235491c8dd83de7f9941689" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 46251 } } peers { permanent_uuid: "ecd1a43fef7d4ba88af942fb70e645a7" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 44791 } }
I20260620 19:09:29.669582 12519 ts_tablet_manager.cc:1428] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260620 19:09:29.669696 12500 heartbeater.cc:503] Master 127.0.203.62:39813 was elected leader, sending a full tablet report...
I20260620 19:09:29.671298 12523 tablet_bootstrap.cc:492] T f3cbf5feb8fd42ee8a82a66695b057f1 P 13989430db72432cba39c31e17d10ad9: Bootstrap starting.
W20260620 19:09:29.671574 12229 tablet.cc:2369] T f3cbf5feb8fd42ee8a82a66695b057f1 P ecd1a43fef7d4ba88af942fb70e645a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:09:29.672384 12523 tablet_bootstrap.cc:654] T f3cbf5feb8fd42ee8a82a66695b057f1 P 13989430db72432cba39c31e17d10ad9: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:29.672691 12523 log.cc:824] T f3cbf5feb8fd42ee8a82a66695b057f1 P 13989430db72432cba39c31e17d10ad9: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:29.676220 12523 tablet_bootstrap.cc:492] T f3cbf5feb8fd42ee8a82a66695b057f1 P 13989430db72432cba39c31e17d10ad9: No bootstrap required, opened a new log
I20260620 19:09:29.676344 12523 ts_tablet_manager.cc:1397] T f3cbf5feb8fd42ee8a82a66695b057f1 P 13989430db72432cba39c31e17d10ad9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 19:09:29.678640 12523 raft_consensus.cc:348] T f3cbf5feb8fd42ee8a82a66695b057f1 P 13989430db72432cba39c31e17d10ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13989430db72432cba39c31e17d10ad9" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36811 } } peers { permanent_uuid: "0c4163f3f235491c8dd83de7f9941689" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 46251 } } peers { permanent_uuid: "ecd1a43fef7d4ba88af942fb70e645a7" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 44791 } }
I20260620 19:09:29.678948 12523 raft_consensus.cc:374] T f3cbf5feb8fd42ee8a82a66695b057f1 P 13989430db72432cba39c31e17d10ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:29.679035 12523 raft_consensus.cc:729] T f3cbf5feb8fd42ee8a82a66695b057f1 P 13989430db72432cba39c31e17d10ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13989430db72432cba39c31e17d10ad9, State: Initialized, Role: FOLLOWER
I20260620 19:09:29.679262 12523 consensus_queue.cc:260] T f3cbf5feb8fd42ee8a82a66695b057f1 P 13989430db72432cba39c31e17d10ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13989430db72432cba39c31e17d10ad9" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36811 } } peers { permanent_uuid: "0c4163f3f235491c8dd83de7f9941689" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 46251 } } peers { permanent_uuid: "ecd1a43fef7d4ba88af942fb70e645a7" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 44791 } }
I20260620 19:09:29.679603 12523 ts_tablet_manager.cc:1428] T f3cbf5feb8fd42ee8a82a66695b057f1 P 13989430db72432cba39c31e17d10ad9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:29.679715 12365 heartbeater.cc:503] Master 127.0.203.62:39813 was elected leader, sending a full tablet report...
I20260620 19:09:29.770888 12524 raft_consensus.cc:482] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:09:29.771046 12524 raft_consensus.cc:504] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13989430db72432cba39c31e17d10ad9" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36811 } } peers { permanent_uuid: "0c4163f3f235491c8dd83de7f9941689" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 46251 } } peers { permanent_uuid: "ecd1a43fef7d4ba88af942fb70e645a7" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 44791 } }
I20260620 19:09:29.771561 12524 leader_election.cc:288] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13989430db72432cba39c31e17d10ad9 (127.0.203.2:36811), ecd1a43fef7d4ba88af942fb70e645a7 (127.0.203.1:44791)
I20260620 19:09:29.775864 12502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:29.771817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:36811 (local address 127.0.203.3:45523)
0620 19:09:29.772233 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:29.772246 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:29.772304 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:29.772531 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:29.772532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:29.772534 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:29.772605 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:29.772608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.773599 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.773600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:29.774810 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:29.774813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.775617 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.775618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:29.775693 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:29.775697 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:09:29.775747 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 19:09:29.775786 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394}
I20260620 19:09:29.776162 12516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:29.772202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:45523 (local address 127.0.203.2:36811)
0620 19:09:29.772311 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:29.772313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:29.772327 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:09:29.772359 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:29.772360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:29.772372 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:29.772433 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:29.772696 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.773474 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.774931 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.775501 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.775573 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:29.775585 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:29.776045 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 19:09:29.776087 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 19:09:29.776435 12319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3cbf5feb8fd42ee8a82a66695b057f1" candidate_uuid: "0c4163f3f235491c8dd83de7f9941689" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13989430db72432cba39c31e17d10ad9" is_pre_election: true
I20260620 19:09:29.776538 12513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:29.772264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:37005 (local address 127.0.203.1:44791)
0620 19:09:29.772495 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:29.772497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:29.772614 (+ 117us) server_negotiation.cc:407] Connection header received
0620 19:09:29.772705 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:29.772706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:29.772717 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:29.772745 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:29.772940 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.773765 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.775295 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.775798 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.775841 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:29.775850 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:29.776424 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 19:09:29.776473 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
I20260620 19:09:29.776604 12319 raft_consensus.cc:2454] T f3cbf5feb8fd42ee8a82a66695b057f1 P 13989430db72432cba39c31e17d10ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4163f3f235491c8dd83de7f9941689 in term 0.
I20260620 19:09:29.776887 12387 leader_election.cc:302] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4163f3f235491c8dd83de7f9941689, 13989430db72432cba39c31e17d10ad9; no voters:
I20260620 19:09:29.776898 12527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:29.771965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:44791 (local address 127.0.203.3:37005)
0620 19:09:29.772555 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:29.772565 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:29.772638 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:29.772805 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:29.772806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:29.772807 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:29.772879 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:29.772880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.773857 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.773857 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:29.775198 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:29.775201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.775920 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.775921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:29.776335 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:29.776339 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:09:29.776772 (+ 433us) client_negotiation.cc:240] Negotiation successful
0620 19:09:29.776813 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":119,"threads_started":1}
I20260620 19:09:29.777060 12524 raft_consensus.cc:2781] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:09:29.777114 12524 raft_consensus.cc:482] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:09:29.777135 12524 raft_consensus.cc:3037] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:29.777319 12182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3cbf5feb8fd42ee8a82a66695b057f1" candidate_uuid: "0c4163f3f235491c8dd83de7f9941689" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecd1a43fef7d4ba88af942fb70e645a7" is_pre_election: true
I20260620 19:09:29.777428 12182 raft_consensus.cc:2454] T f3cbf5feb8fd42ee8a82a66695b057f1 P ecd1a43fef7d4ba88af942fb70e645a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4163f3f235491c8dd83de7f9941689 in term 0.
I20260620 19:09:29.778005 12524 raft_consensus.cc:504] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13989430db72432cba39c31e17d10ad9" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36811 } } peers { permanent_uuid: "0c4163f3f235491c8dd83de7f9941689" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 46251 } } peers { permanent_uuid: "ecd1a43fef7d4ba88af942fb70e645a7" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 44791 } }
I20260620 19:09:29.778136 12524 leader_election.cc:288] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [CANDIDATE]: Term 1 election: Requested vote from peers 13989430db72432cba39c31e17d10ad9 (127.0.203.2:36811), ecd1a43fef7d4ba88af942fb70e645a7 (127.0.203.1:44791)
I20260620 19:09:29.778316 12182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3cbf5feb8fd42ee8a82a66695b057f1" candidate_uuid: "0c4163f3f235491c8dd83de7f9941689" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecd1a43fef7d4ba88af942fb70e645a7"
I20260620 19:09:29.778381 12319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3cbf5feb8fd42ee8a82a66695b057f1" candidate_uuid: "0c4163f3f235491c8dd83de7f9941689" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13989430db72432cba39c31e17d10ad9"
I20260620 19:09:29.778398 12182 raft_consensus.cc:3037] T f3cbf5feb8fd42ee8a82a66695b057f1 P ecd1a43fef7d4ba88af942fb70e645a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:29.778450 12319 raft_consensus.cc:3037] T f3cbf5feb8fd42ee8a82a66695b057f1 P 13989430db72432cba39c31e17d10ad9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:29.779263 12319 raft_consensus.cc:2454] T f3cbf5feb8fd42ee8a82a66695b057f1 P 13989430db72432cba39c31e17d10ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4163f3f235491c8dd83de7f9941689 in term 1.
I20260620 19:09:29.779294 12182 raft_consensus.cc:2454] T f3cbf5feb8fd42ee8a82a66695b057f1 P ecd1a43fef7d4ba88af942fb70e645a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4163f3f235491c8dd83de7f9941689 in term 1.
I20260620 19:09:29.779426 12387 leader_election.cc:302] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4163f3f235491c8dd83de7f9941689, 13989430db72432cba39c31e17d10ad9; no voters:
I20260620 19:09:29.779501 12524 raft_consensus.cc:2781] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:29.779623 12524 raft_consensus.cc:686] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [term 1 LEADER]: Becoming Leader. State: Replica: 0c4163f3f235491c8dd83de7f9941689, State: Running, Role: LEADER
I20260620 19:09:29.779713 12524 consensus_queue.cc:237] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13989430db72432cba39c31e17d10ad9" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36811 } } peers { permanent_uuid: "0c4163f3f235491c8dd83de7f9941689" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 46251 } } peers { permanent_uuid: "ecd1a43fef7d4ba88af942fb70e645a7" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 44791 } }
I20260620 19:09:29.780535 12016 catalog_manager.cc:5495] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c4163f3f235491c8dd83de7f9941689 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "0c4163f3f235491c8dd83de7f9941689" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13989430db72432cba39c31e17d10ad9" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c4163f3f235491c8dd83de7f9941689" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 46251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecd1a43fef7d4ba88af942fb70e645a7" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 44791 } health_report { overall_health: UNKNOWN } } }
I20260620 19:09:29.797977 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:29.794858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36464 (local address 127.0.203.3:46251)
0620 19:09:29.794899 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:29.794900 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:29.794909 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:09:29.794980 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:29.794981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:29.795003 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:29.795080 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:09:29.795255 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.795951 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.797069 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:29.797351 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:29.797417 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:29.797613 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:09:29.797736 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:09:29.797826 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:29.797884 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 19:09:29.797904 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:09:29.800057 12319 raft_consensus.cc:1261] T f3cbf5feb8fd42ee8a82a66695b057f1 P 13989430db72432cba39c31e17d10ad9 [term 1 FOLLOWER]: Refusing update from remote peer 0c4163f3f235491c8dd83de7f9941689: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:09:29.800081 12182 raft_consensus.cc:1261] T f3cbf5feb8fd42ee8a82a66695b057f1 P ecd1a43fef7d4ba88af942fb70e645a7 [term 1 FOLLOWER]: Refusing update from remote peer 0c4163f3f235491c8dd83de7f9941689: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:09:29.800366 12524 consensus_queue.cc:1035] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13989430db72432cba39c31e17d10ad9" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:09:29.800506 12524 consensus_queue.cc:1035] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecd1a43fef7d4ba88af942fb70e645a7" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 44791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:09:29.803776 12531 mvcc.cc:205] Tried to move back new op lower bound from 7299000605897707520 to 7299000605818118144. Current Snapshot: MvccSnapshot[applied={T|T < 7299000605897707520}]
I20260620 19:09:29.804229 12533 mvcc.cc:205] Tried to move back new op lower bound from 7299000605897707520 to 7299000605818118144. Current Snapshot: MvccSnapshot[applied={T|T < 7299000605897707520 or (T in {7299000605897707520})}]
I20260620 19:09:29.809104 12532 mvcc.cc:205] Tried to move back new op lower bound from 7299000605897707520 to 7299000605818118144. Current Snapshot: MvccSnapshot[applied={T|T < 7299000605897707520 or (T in {7299000605897707520})}]
I20260620 19:09:29.811654 12016 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:09:29.811760 12016 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:09:29.812812 12016 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7dbb5f1c7324792a928e8300de4cb69: Sending DeleteTablet for 3 replicas of tablet f3cbf5feb8fd42ee8a82a66695b057f1
I20260620 19:09:29.813360 12299 tablet_service.cc:1514] Processing DeleteTablet for tablet f3cbf5feb8fd42ee8a82a66695b057f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:29 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:40304
I20260620 19:09:29.813578 12423 tablet_service.cc:1514] Processing DeleteTablet for tablet f3cbf5feb8fd42ee8a82a66695b057f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:29 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:36460
I20260620 19:09:29.813789 12153 tablet_service.cc:1514] Processing DeleteTablet for tablet f3cbf5feb8fd42ee8a82a66695b057f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:09:29 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:39920
I20260620 19:09:29.815549 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 12097
I20260620 19:09:29.823230 12541 tablet_replica.cc:330] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689: stopping tablet replica
I20260620 19:09:29.823369 12541 raft_consensus.cc:2229] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:09:29.823504 12541 raft_consensus.cc:2258] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:09:29.823988 12541 ts_tablet_manager.cc:1905] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:09:29.825628 12541 ts_tablet_manager.cc:1918] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:09:29.825699 12541 log.cc:1196] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/f3cbf5feb8fd42ee8a82a66695b057f1
I20260620 19:09:29.825949 12541 ts_tablet_manager.cc:1939] T f3cbf5feb8fd42ee8a82a66695b057f1 P 0c4163f3f235491c8dd83de7f9941689: Deleting consensus metadata
I20260620 19:09:29.826380 12000 catalog_manager.cc:4841] TS 0c4163f3f235491c8dd83de7f9941689 (127.0.203.3:46251): tablet f3cbf5feb8fd42ee8a82a66695b057f1 (table test-table [id=dc7625d4827c4c21a08927f64bc127a1]) successfully deleted
W20260620 19:09:29.835845 12001 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.1:44791 (error 108)
I20260620 19:09:29.842372 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 12234
I20260620 19:09:29.847281 12515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:29.843458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:44791 (local address 127.0.0.1:39932)
0620 19:09:29.847174 (+ 3716us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:29.847238 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.1:44791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3684}
Jun 20 19:09:29 dist-test-slave-wd30 krb5kdc[11540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
W20260620 19:09:29.848114 12001 catalog_manager.cc:4584] TS ecd1a43fef7d4ba88af942fb70e645a7 (127.0.203.1:44791): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3cbf5feb8fd42ee8a82a66695b057f1: Network error: Client connection negotiation failed: client connection to 127.0.203.1:44791: connect: Connection refused (error 111)
W20260620 19:09:29.850940 12515 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:09:29.850691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:36811 (local address 127.0.0.1:40310)
0620 19:09:29.850735 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:29.850745 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:29.850783 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:29.850894 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.2:36811: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 19:09:29.855263 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 12371
I20260620 19:09:29.862107 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 11984
I20260620 19:09:29.867408 812 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 19:09:30.202219 812 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 19:09:30.202342 812 mini_ranger.cc:66] Stopping Ranger...
I20260620 19:09:30.547384 812 mini_ranger.cc:68] Stopped Ranger
2026-06-20 19:09:30.547 UTC [11570] LOG: received smart shutdown request
2026-06-20 19:09:30.550 UTC [11570] LOG: background worker "logical replication launcher" (PID 11579) exited with exit code 1
2026-06-20 19:09:30.551 UTC [11574] LOG: shutting down
2026-06-20 19:09:30.706 UTC [11570] LOG: database system is shut down
2026-06-20T19:09:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47076 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:30 dist-test-slave-wd30 krb5kdc[12552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:30 dist-test-slave-wd30 krb5kdc[12552](info): set up 2 sockets
Jun 20 19:09:30 dist-test-slave-wd30 krb5kdc[12552](info): commencing operation
krb5kdc: starting...
W20260620 19:09:31.855017 812 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:09:31 dist-test-slave-wd30 krb5kdc[12552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:31Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.203.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.203.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.203.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/rangeradmin_127.0.203.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.203.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/rangeradmin_127.0.203.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.203.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.203.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.203.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/rangerlookup_127.0.203.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.203.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/rangerlookup_127.0.203.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.203.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.203.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.203.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/HTTP_127.0.203.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.203.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/HTTP_127.0.203.20@KRBTEST.COM.keytab.
I20260620 19:09:32.018083 812 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskeuOXoM/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/postgres -l logfile start
2026-06-20 19:09:34.521 UTC [12582] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 19:09:34.521 UTC [12582] LOG: listening on IPv4 address "127.0.203.20", port 40127
2026-06-20 19:09:34.524 UTC [12582] LOG: listening on Unix socket "/tmp/.s.PGSQL.40127"
2026-06-20 19:09:34.546 UTC [12585] LOG: database system was shut down at 2026-06-20 19:09:33 UTC
2026-06-20 19:09:34.550 UTC [12582] LOG: database system is ready to accept connections
I20260620 19:09:35.502812 812 mini_postgres.cc:96] Postgres bound to 40127
2026-06-20 19:09:35.508 UTC [12593] FATAL: database "slave" does not exist
127.0.203.20:40127 - accepting connections
I20260620 19:09:35.509222 812 mini_ranger.cc:161] Starting Ranger...
I20260620 19:09:35.524168 812 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 19:09:36.000151 812 mini_ranger.cc:87] Created ranger Postgres database
I20260620 19:09:36.000221 812 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ranger-admin
2026-06-20 19:09:36,288 [I] DB FLAVOR :POSTGRES
2026-06-20 19:09:36,289 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:09:36,289 [I] Checking connection..
2026-06-20 19:09:36,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 19:09:36,562 [I] Checking connection passed.
2026-06-20 19:09:36,562 [I] --------- Verifying version history table ---------
2026-06-20 19:09:36,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:09:36,811 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:09:36,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:09:37,056 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:09:37,057 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 19:09:37,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 19:09:37.290 UTC [12657] WARNING: there is no transaction in progress
2026-06-20 19:09:37,296 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 19:09:37,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:09:37,550 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 19:09:37,551 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:09:37,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 19:09:37,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 19:09:38,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wd30.c.gcp-upstream.internal','N') ;"
2026-06-20 19:09:38,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:09:38,495 [I] Table x_portal_user does not exist in database ranger
2026-06-20 19:09:38,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:09:38,744 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 19:09:38,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:09:39,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:09:39,234 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 19:09:39,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 19:09:40.230 UTC [12765] WARNING: there is no transaction in progress
2026-06-20 19:09:40.249 UTC [12765] WARNING: there is no transaction in progress
2026-06-20 19:09:40.270 UTC [12765] WARNING: there is no transaction in progress
2026-06-20 19:09:40.285 UTC [12765] WARNING: there is no transaction in progress
2026-06-20 19:09:40.302 UTC [12765] WARNING: there is no transaction in progress
2026-06-20 19:09:40.321 UTC [12765] WARNING: there is no transaction in progress
2026-06-20 19:09:40.381 UTC [12765] WARNING: there is no transaction in progress
2026-06-20 19:09:40.388 UTC [12765] WARNING: there is no transaction in progress
2026-06-20 19:09:40.402 UTC [12765] WARNING: there is no transaction in progress
2026-06-20 19:09:40.411 UTC [12765] WARNING: there is no transaction in progress
2026-06-20 19:09:40.422 UTC [12765] WARNING: there is no transaction in progress
2026-06-20 19:09:40.438 UTC [12765] WARNING: there is no transaction in progress
2026-06-20 19:09:40.448 UTC [12765] WARNING: there is no transaction in progress
2026-06-20 19:09:40.454 UTC [12765] WARNING: there is no transaction in progress
2026-06-20 19:09:40.462 UTC [12765] WARNING: there is no transaction in progress
2026-06-20 19:09:40.470 UTC [12765] WARNING: there is no transaction in progress
2026-06-20 19:09:41,151 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 19:09:41,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 19:09:41,383 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 19:09:41,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:09:41,643 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 19:09:41,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:09:41,902 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 19:09:41,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:09:42,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:09:42,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wd30.c.gcp-upstream.internal';"
2026-06-20 19:09:42,660 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 19:09:42,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 19:09:42,891 [I] DB_PATCHES have already been applied
I20260620 19:09:42.896479 812 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ranger-admin:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskeuOXoM/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 19:09:42.896533 812 mini_ranger.cc:193] Using host: 127.0.203.20
I20260620 19:09:42.899508 812 mini_ranger.cc:227] Ranger admin URL: http://127.0.203.20:53325
Jun 20, 2026 7:09:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:09:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:09:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 7:09:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 7:09:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 7:09:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 7:09:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 19:09:44.001153 812 mini_ranger.cc:160] Time spent starting Ranger: real 8.492s user 0.000s sys 0.008s
Jun 20, 2026 7:10:04 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:10:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 7:10:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:10:04 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:10:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:10:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:10:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:10:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:10:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:10:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:10:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:10:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:10:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:10:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:10:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:10:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:10:04 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:10:05 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 19:10:05.965862 812 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.203.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.203.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.203.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/rangerkms_127.0.203.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.203.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/rangerkms_127.0.203.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.203.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.203.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.203.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/HTTP_127.0.203.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.203.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/HTTP_127.0.203.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 19:10:06 dist-test-slave-wd30 krb5kdc[12552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982606, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 19:10:06.046087 812 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 19:10:06.055796 812 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 19:10:06.954668 812 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 19:10:06.954754 812 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ranger-kms
I20260620 19:10:06.954777 812 mini_ranger_kms.cc:227] Using postgres at 127.0.203.20:40127
2026-06-20 19:10:07,113 [I] DB FLAVOR :POSTGRES
2026-06-20 19:10:07,114 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:10:07,114 [I] Checking connection
2026-06-20 19:10:07,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 19:10:07,341 [I] connection success
2026-06-20 19:10:07,341 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 19:10:07,342 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 19:10:07,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 19:10:07,608 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 19:10:07,608 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:10:07,608 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 19:10:07,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.20:40127/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 19:10:07,885 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 19:10:08.367372 812 mini_ranger_kms.cc:310] Created kms service
I20260620 19:10:08.578365 812 mini_ranger_kms.cc:326] Created kudu user
I20260620 19:10:08.626124 812 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 19:10:08 dist-test-slave-wd30 krb5kdc[12552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982606, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
I20260620 19:10:09.069067 812 mini_ranger_kms.cc:384] Added ranger policy
I20260620 19:10:09.069271 812 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ranger-kms:/tmp/dist-test-taskeuOXoM/build/release/bin/postgresql.jar:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskeuOXoM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 19:10:09.069294 812 mini_ranger_kms.cc:240] Using host: 127.0.203.20
I20260620 19:10:09.073230 812 mini_ranger_kms.cc:276] Ranger KMS PID: 12945
I20260620 19:10:09.073338 812 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.203.20:37927
Jun 20, 2026 7:10:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:10:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:10:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:10:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:10:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:10:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 19:10:11.342849 812 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.297s user 0.001s sys 0.142s
I20260620 19:10:11.342947 812 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 19:10:11.342991 812 mini_ranger_kms.cc:401] 127.0.203.20:37927/kms/v1/keys
Jun 20 19:10:11 dist-test-slave-wd30 krb5kdc[12552](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982611, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.203.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 19:10:11 dist-test-slave-wd30 krb5kdc[12552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982611, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.203.20@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
Jun 20 19:10:12 dist-test-slave-wd30 krb5kdc[12552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982611, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.203.20@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
Jun 20 19:10:14 dist-test-slave-wd30 krb5kdc[12552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:10:14.606891 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:44563
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:44563
--ranger_config_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.20:37927/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:14.691835 12996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:14.692055 12996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:14.692077 12996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:14.693291 12996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:14.693342 12996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:14.693357 12996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:14.693369 12996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:14.693382 12996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:14.695035 12996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.203.20:37927/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:44563
--ranger_config_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:44563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:14.695335 12996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:14.695580 12996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:14.700696 13001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:14.700685 13004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:14.700721 13002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:14.700986 12996 server_base.cc:1034] running on GCE node
I20260620 19:10:14.701392 12996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:14.701786 12996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:14.702960 12996 hybrid_clock.cc:648] HybridClock initialized: now 1781982614702934 us; error 45 us; skew 500 ppm
Jun 20 19:10:14 dist-test-slave-wd30 krb5kdc[12552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:14.708969 12996 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:10:14.709473 12996 webserver.cc:460] Webserver started at http://127.0.203.62:44265/ using document root <none> and password file <none>
I20260620 19:10:14.709714 12996 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:14.709775 12996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:14.709935 12996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:10:14 dist-test-slave-wd30 krb5kdc[12552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
I20260620 19:10:14.774008 12996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "954ab55185a24a0d85e3a95878dd3702"
format_stamp: "Formatted at 2026-06-20 19:10:14 on dist-test-slave-wd30"
server_key: "72defdada34cf5447be9057b8b64371f"
server_key_iv: "57704a456139b2e3c10af0a74f9dcced"
server_key_version: "kuduclusterkey@0"
I20260620 19:10:14.774591 12996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "954ab55185a24a0d85e3a95878dd3702"
format_stamp: "Formatted at 2026-06-20 19:10:14 on dist-test-slave-wd30"
server_key: "72defdada34cf5447be9057b8b64371f"
server_key_iv: "57704a456139b2e3c10af0a74f9dcced"
server_key_version: "kuduclusterkey@0"
I20260620 19:10:14.776258 12996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:10:14.791464 13011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:14.791671 12996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:10:14.791761 12996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "954ab55185a24a0d85e3a95878dd3702"
format_stamp: "Formatted at 2026-06-20 19:10:14 on dist-test-slave-wd30"
server_key: "72defdada34cf5447be9057b8b64371f"
server_key_iv: "57704a456139b2e3c10af0a74f9dcced"
server_key_version: "kuduclusterkey@0"
I20260620 19:10:14.791863 12996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:14.810452 12996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:14.812693 12996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:14.812861 12996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:14.817822 12996 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:44563
I20260620 19:10:14.817898 13063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:44563 every 8 connection(s)
I20260620 19:10:14.818260 12996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:14.818977 13064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:14.821297 13064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702: Bootstrap starting.
I20260620 19:10:14.821898 13064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:14.822213 13064 log.cc:824] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:14.822954 13064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702: No bootstrap required, opened a new log
I20260620 19:10:14.823589 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 12996
I20260620 19:10:14.823695 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:10:14.823865 812 external_mini_cluster.cc:1428] Setting key 58f4d7878966df6e51c32f51a14e1d35
I20260620 19:10:14.825481 13064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "954ab55185a24a0d85e3a95878dd3702" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44563 } }
I20260620 19:10:14.825651 13064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:14.825714 13064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954ab55185a24a0d85e3a95878dd3702, State: Initialized, Role: FOLLOWER
I20260620 19:10:14.825863 13064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "954ab55185a24a0d85e3a95878dd3702" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44563 } }
I20260620 19:10:14.825963 13064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:14.825999 13064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:14.826128 13064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:14.826871 13064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "954ab55185a24a0d85e3a95878dd3702" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44563 } }
I20260620 19:10:14.826998 13064 leader_election.cc:302] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 954ab55185a24a0d85e3a95878dd3702; no voters:
I20260620 19:10:14.827211 13064 leader_election.cc:288] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:14.827375 13070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:14.827481 13064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:14.827620 13070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [term 1 LEADER]: Becoming Leader. State: Replica: 954ab55185a24a0d85e3a95878dd3702, State: Running, Role: LEADER
I20260620 19:10:14.827759 13070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "954ab55185a24a0d85e3a95878dd3702" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44563 } }
I20260620 19:10:14.828161 13070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "954ab55185a24a0d85e3a95878dd3702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "954ab55185a24a0d85e3a95878dd3702" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44563 } } }
I20260620 19:10:14.828305 13070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:14.828473 13070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 954ab55185a24a0d85e3a95878dd3702. Latest consensus state: current_term: 1 leader_uuid: "954ab55185a24a0d85e3a95878dd3702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "954ab55185a24a0d85e3a95878dd3702" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44563 } } }
I20260620 19:10:14.828613 13070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:14.828770 13078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:14.828915 13064 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 19:10:14.829349 13078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 19:10:14 dist-test-slave-wd30 krb5kdc[12552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:14.835426 13078 catalog_manager.cc:1347] Generated new cluster ID: 858db870de3e4a1283fb892f18dc2732
I20260620 19:10:14.835597 13078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:14.843102 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:14.824414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40068 (local address 127.0.203.62:44563)
0620 19:10:14.824594 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:14.824604 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:14.824634 (+ 30us) server_negotiation.cc:407] Connection header received
0620 19:10:14.825492 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:14.825502 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:14.825572 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:14.825722 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:14.826659 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:14.828120 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:14.829226 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:14.829815 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:14.829919 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:14.833806 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:14.833850 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:14.833850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:14.833965 (+ 115us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:14.840500 (+ 6535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:14.840981 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:14.840984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:14.840985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:14.841120 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:14.841352 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:14.841352 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:14.841353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:14.841757 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:14.842357 (+ 600us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:14.842433 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 19:10:14.842782 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
I20260620 19:10:14.849220 13078 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:14.850031 13078 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:14.853713 13078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702: Generated new TSK 0
I20260620 19:10:14.853972 13078 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
I20260620 19:10:15.605826 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.20:37927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44563
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:15.717597 13109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:15.717830 13109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:15.717872 13109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:15.719223 13109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:15.719282 13109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:15.719326 13109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:10:15.720855 13109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.203.20:37927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:15.721127 13109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:15.721354 13109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:15.726518 13115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:15.726764 13117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:15.726826 13114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:15.727023 13109 server_base.cc:1034] running on GCE node
I20260620 19:10:15.727468 13109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:15.727708 13109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:15.735055 13109 hybrid_clock.cc:648] HybridClock initialized: now 1781982615735020 us; error 57 us; skew 500 ppm
Jun 20 19:10:15 dist-test-slave-wd30 krb5kdc[12552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:15.737932 13109 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:10:15.738480 13109 webserver.cc:460] Webserver started at http://127.0.203.1:41623/ using document root <none> and password file <none>
I20260620 19:10:15.738739 13109 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:15.738821 13109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:15.738978 13109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:10:15 dist-test-slave-wd30 krb5kdc[12552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
I20260620 19:10:15.752696 13109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "d3a9894a527c4fdb958d8f71c6903f2c"
format_stamp: "Formatted at 2026-06-20 19:10:15 on dist-test-slave-wd30"
server_key: "1d59a094398540340fbcbb05b11429d4"
server_key_iv: "e8d449054c54dd18f680c2d141b05858"
server_key_version: "kuduclusterkey@0"
I20260620 19:10:15.753181 13109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "d3a9894a527c4fdb958d8f71c6903f2c"
format_stamp: "Formatted at 2026-06-20 19:10:15 on dist-test-slave-wd30"
server_key: "1d59a094398540340fbcbb05b11429d4"
server_key_iv: "e8d449054c54dd18f680c2d141b05858"
server_key_version: "kuduclusterkey@0"
I20260620 19:10:15.754845 13109 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:15.782915 13126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:15.783182 13109 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260620 19:10:15.783288 13109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "d3a9894a527c4fdb958d8f71c6903f2c"
format_stamp: "Formatted at 2026-06-20 19:10:15 on dist-test-slave-wd30"
server_key: "1d59a094398540340fbcbb05b11429d4"
server_key_iv: "e8d449054c54dd18f680c2d141b05858"
server_key_version: "kuduclusterkey@0"
I20260620 19:10:15.783424 13109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:15.803854 13109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:15.806205 13109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:15.806393 13109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:15.806711 13109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:15.807144 13109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:15.807233 13109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:15.807328 13109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:15.807401 13109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:15.815608 13109 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:42131
I20260620 19:10:15.816154 13109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:10:15.818203 13239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:42131 every 8 connection(s)
I20260620 19:10:15.825515 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 13109
I20260620 19:10:15.825721 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:10:15.825909 812 external_mini_cluster.cc:1428] Setting key 37738abe13af6a1e2596912f9b3e03fe
Jun 20 19:10:15 dist-test-slave-wd30 krb5kdc[12552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:15.834394 13243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:15.818839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:40675 (local address 127.0.203.62:44563)
0620 19:10:15.824414 (+ 5575us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:15.824416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:15.824424 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:10:15.824455 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:15.824456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:15.824469 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:15.824533 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:15.825345 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:15.826130 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:15.827641 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:15.827988 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:15.828053 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:15.830795 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:15.830797 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:15.830798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:15.830805 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:15.832689 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:15.833194 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:15.833196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:15.833196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:15.833260 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:15.833568 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:15.833569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:15.833569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:15.833707 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:15.833771 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:15.834166 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 19:10:15.834266 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5541,"thread_start_us":92,"threads_started":1}
I20260620 19:10:15.834519 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:15.818798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:44563 (local address 127.0.203.1:40675)
0620 19:10:15.823479 (+ 4681us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:15.823539 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:15.824215 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:15.824674 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:15.824680 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:15.824948 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:15.825231 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:15.825238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:15.826237 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:15.826238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:15.827546 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:15.827549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:15.828138 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:15.828139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:15.828273 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:15.828941 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:15.828952 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:15.830682 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:15.832784 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:15.832791 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:15.832793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:15.833101 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:15.833331 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:15.833332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:15.833333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:15.833488 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:15.833801 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:15.833809 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:15.833985 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 19:10:15.834106 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 19:10:15.834264 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4624,"thread_start_us":98,"threads_started":1}
I20260620 19:10:15.835208 13240 heartbeater.cc:347] Connected to a master server at 127.0.203.62:44563
I20260620 19:10:15.835319 13240 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:15.835523 13240 heartbeater.cc:511] Master 127.0.203.62:44563 requested a full tablet report, sending...
I20260620 19:10:15.836161 13028 ts_manager.cc:194] Registered new tserver with Master: d3a9894a527c4fdb958d8f71c6903f2c (127.0.203.1:42131)
WARNING: no policy specified for kudu/127.0.203.2@KRBTEST.COM; defaulting to no policy
I20260620 19:10:15.837121 13028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:40675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
I20260620 19:10:15.860396 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.20:37927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44563
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:10:15 dist-test-slave-wd30 krb5kdc[12552](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 19:10:15.992115 13246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:15.992452 13246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:15.992493 13246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:15.994531 13246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:15.994669 13246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:15.994771 13246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:10:15.997390 13246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.203.20:37927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:15.997922 13246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:15.998294 13246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:16.005601 13254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:16.005601 13253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:16.006050 13256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:16.007194 13246 server_base.cc:1034] running on GCE node
I20260620 19:10:16.007555 13246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:16.007860 13246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:16.009043 13246 hybrid_clock.cc:648] HybridClock initialized: now 1781982616009016 us; error 43 us; skew 500 ppm
Jun 20 19:10:16 dist-test-slave-wd30 krb5kdc[12552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:16.012706 13246 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:10:16.013365 13246 webserver.cc:460] Webserver started at http://127.0.203.2:44601/ using document root <none> and password file <none>
I20260620 19:10:16.013738 13246 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:16.013856 13246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:16.014022 13246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:10:16 dist-test-slave-wd30 krb5kdc[12552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
I20260620 19:10:16.028088 13246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "d3cc2a7f85274ddb99f1fe8507f28051"
format_stamp: "Formatted at 2026-06-20 19:10:16 on dist-test-slave-wd30"
server_key: "02a4912ef5d6997cb0ff38984519c1fd"
server_key_iv: "cde4ed098a4d15883dea50084b134070"
server_key_version: "kuduclusterkey@0"
I20260620 19:10:16.028599 13246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "d3cc2a7f85274ddb99f1fe8507f28051"
format_stamp: "Formatted at 2026-06-20 19:10:16 on dist-test-slave-wd30"
server_key: "02a4912ef5d6997cb0ff38984519c1fd"
server_key_iv: "cde4ed098a4d15883dea50084b134070"
server_key_version: "kuduclusterkey@0"
I20260620 19:10:16.030351 13246 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:16.057425 13263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:16.057713 13246 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.001s
I20260620 19:10:16.057857 13246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "d3cc2a7f85274ddb99f1fe8507f28051"
format_stamp: "Formatted at 2026-06-20 19:10:16 on dist-test-slave-wd30"
server_key: "02a4912ef5d6997cb0ff38984519c1fd"
server_key_iv: "cde4ed098a4d15883dea50084b134070"
server_key_version: "kuduclusterkey@0"
I20260620 19:10:16.057979 13246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:16.073007 13246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:16.075433 13246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:16.075716 13246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:16.076021 13246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:16.076457 13246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:16.076517 13246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:16.076588 13246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:16.076622 13246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:16.084472 13246 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:45867
I20260620 19:10:16.085044 13246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:10:16.086743 13377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:45867 every 8 connection(s)
Jun 20 19:10:16 dist-test-slave-wd30 krb5kdc[12552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:16.093360 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 13246
I20260620 19:10:16.093470 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:10:16.093714 812 external_mini_cluster.cc:1428] Setting key 288ebb04dffcb3569ad512b26f33ebd7
I20260620 19:10:16.097712 13243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:16.086319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:34307 (local address 127.0.203.62:44563)
0620 19:10:16.086382 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:16.086385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:16.087219 (+ 834us) server_negotiation.cc:407] Connection header received
0620 19:10:16.088031 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:16.088033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:16.088048 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:16.088084 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:16.089024 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.089619 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.090798 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.091191 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.091283 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:16.093671 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:16.093673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:16.093674 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:16.093681 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:16.095704 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:16.096241 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:16.096243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:16.096244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:16.096325 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:16.096728 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:16.096730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:16.096730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:16.096892 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:16.096941 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:16.097410 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 19:10:16.097527 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 19:10:16.098660 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:16.085967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:44563 (local address 127.0.203.2:34307)
0620 19:10:16.087040 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:16.087089 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:16.087877 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:16.088300 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:16.088305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:16.088586 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:16.088902 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:16.088908 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.089752 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.089753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:16.090686 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:16.090688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.091295 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.091296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:16.091432 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:16.092069 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:16.092086 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:16.093558 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:16.095807 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:16.095814 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:16.095817 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:16.096124 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:16.096413 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:16.096414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:16.096414 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:16.096618 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:16.096996 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:16.097003 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:16.097210 (+ 207us) client_negotiation.cc:769] Sending connection context
0620 19:10:16.098125 (+ 915us) client_negotiation.cc:240] Negotiation successful
0620 19:10:16.098338 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"thread_start_us":91,"threads_started":1}
I20260620 19:10:16.099473 13378 heartbeater.cc:347] Connected to a master server at 127.0.203.62:44563
I20260620 19:10:16.099630 13378 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:16.099895 13378 heartbeater.cc:511] Master 127.0.203.62:44563 requested a full tablet report, sending...
I20260620 19:10:16.100385 13028 ts_manager.cc:194] Registered new tserver with Master: d3cc2a7f85274ddb99f1fe8507f28051 (127.0.203.2:45867)
I20260620 19:10:16.100988 13028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:34307
WARNING: no policy specified for kudu/127.0.203.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
I20260620 19:10:16.122218 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.20:37927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44563
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:16.242614 13383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:16.243139 13383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:16.243188 13383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:16.245054 13383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:16.245139 13383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:16.245190 13383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:10:16.246956 13383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.203.20:37927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:16.247294 13383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:16.247562 13383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:16.252676 13388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:16.252699 13389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:16.252723 13391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:16.253028 13383 server_base.cc:1034] running on GCE node
I20260620 19:10:16.253239 13383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:16.253548 13383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:16.254706 13383 hybrid_clock.cc:648] HybridClock initialized: now 1781982616254696 us; error 28 us; skew 500 ppm
Jun 20 19:10:16 dist-test-slave-wd30 krb5kdc[12552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:16.257259 13383 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:10:16.257805 13383 webserver.cc:460] Webserver started at http://127.0.203.3:44651/ using document root <none> and password file <none>
I20260620 19:10:16.258051 13383 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:16.258096 13383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:16.258262 13383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:10:16 dist-test-slave-wd30 krb5kdc[12552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
I20260620 19:10:16.271778 13383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "648908d936904f88a368ebf753262fec"
format_stamp: "Formatted at 2026-06-20 19:10:16 on dist-test-slave-wd30"
server_key: "0fcd65b69d21be609f19da2e2995e2c6"
server_key_iv: "c9fa89ff34d3a1d067920ed8adf26ed9"
server_key_version: "kuduclusterkey@0"
I20260620 19:10:16.272475 13383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "648908d936904f88a368ebf753262fec"
format_stamp: "Formatted at 2026-06-20 19:10:16 on dist-test-slave-wd30"
server_key: "0fcd65b69d21be609f19da2e2995e2c6"
server_key_iv: "c9fa89ff34d3a1d067920ed8adf26ed9"
server_key_version: "kuduclusterkey@0"
I20260620 19:10:16.283720 13383 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.002s sys 0.000s
I20260620 19:10:16.299703 13398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:16.299995 13383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:10:16.300104 13383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "648908d936904f88a368ebf753262fec"
format_stamp: "Formatted at 2026-06-20 19:10:16 on dist-test-slave-wd30"
server_key: "0fcd65b69d21be609f19da2e2995e2c6"
server_key_iv: "c9fa89ff34d3a1d067920ed8adf26ed9"
server_key_version: "kuduclusterkey@0"
I20260620 19:10:16.300226 13383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:16.341611 13383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:16.344353 13383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:16.344594 13383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:16.344879 13383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:16.345351 13383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:16.345441 13383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:16.345515 13383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:16.345572 13383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:16.353238 13383 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:37513
I20260620 19:10:16.353801 13383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:16.355830 13511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:37513 every 8 connection(s)
I20260620 19:10:16.359304 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 13383
I20260620 19:10:16.359400 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:10:16.359567 812 external_mini_cluster.cc:1428] Setting key 25e74f9cb70b944ab533f00403bfc8ec
Jun 20 19:10:16 dist-test-slave-wd30 krb5kdc[12552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:16.368491 13243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:16.356784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:59641 (local address 127.0.203.62:44563)
0620 19:10:16.356832 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:16.356834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:16.357109 (+ 275us) server_negotiation.cc:407] Connection header received
0620 19:10:16.357875 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:16.357877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:16.357890 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:16.357923 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:16.358788 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.359558 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.361898 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.362190 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.362238 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:16.364943 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:16.364944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:16.364945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:16.364953 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:16.366776 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:16.367332 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:16.367333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:16.367334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:16.367394 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:16.367700 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:16.367701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:16.367702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:16.367843 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:16.367889 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:16.368301 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 19:10:16.368384 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:10:16.369068 13514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:16.356483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:44563 (local address 127.0.203.3:59641)
0620 19:10:16.356979 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:16.357022 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:16.357744 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:16.358118 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:16.358123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:16.358378 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:16.358660 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:16.358666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.359705 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.359707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:16.361755 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:16.361758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.362349 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.362350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:16.362492 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:16.363162 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:16.363173 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:16.364828 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:16.366876 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:16.366884 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:16.366886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:16.367237 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:16.367472 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:16.367473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:16.367473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:16.367606 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:16.367938 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:16.367945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:16.368135 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 19:10:16.368634 (+ 499us) client_negotiation.cc:240] Negotiation successful
0620 19:10:16.368840 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":82,"threads_started":1}
I20260620 19:10:16.369629 13512 heartbeater.cc:347] Connected to a master server at 127.0.203.62:44563
I20260620 19:10:16.369737 13512 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:16.369923 13512 heartbeater.cc:511] Master 127.0.203.62:44563 requested a full tablet report, sending...
I20260620 19:10:16.371163 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:10:16.371881 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 13109
I20260620 19:10:16.370471 13028 ts_manager.cc:194] Registered new tserver with Master: 648908d936904f88a368ebf753262fec (127.0.203.3:37513)
I20260620 19:10:16.373716 13028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:59641
I20260620 19:10:16.379029 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 13246
I20260620 19:10:16.384454 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 13383
I20260620 19:10:16.390373 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 12996
I20260620 19:10:16.397779 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:44563
--webserver_interface=127.0.203.62
--webserver_port=44265
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:44563
--ranger_config_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.20:37927/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.203.20:37927/kms with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:16.484280 13515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:16.484534 13515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:16.484573 13515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:16.485723 13515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:16.485795 13515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:16.485822 13515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:16.485853 13515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:16.485888 13515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:16.487563 13515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.203.20:37927/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:44563
--ranger_config_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:44563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=44265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:16.487847 13515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:16.488102 13515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:16.493170 13521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:16.493266 13520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:16.493183 13523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:16.493285 13515 server_base.cc:1034] running on GCE node
I20260620 19:10:16.493634 13515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:16.493968 13515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:16.495185 13515 hybrid_clock.cc:648] HybridClock initialized: now 1781982616495163 us; error 37 us; skew 500 ppm
Jun 20 19:10:16 dist-test-slave-wd30 krb5kdc[12552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:16.497772 13515 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:10:16.498294 13515 webserver.cc:460] Webserver started at http://127.0.203.62:44265/ using document root <none> and password file <none>
I20260620 19:10:16.498564 13515 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:16.498663 13515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:10:16 dist-test-slave-wd30 krb5kdc[12552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
I20260620 19:10:16.582849 13515 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:16.583913 13531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:16.584108 13515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:16.584229 13515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "954ab55185a24a0d85e3a95878dd3702"
format_stamp: "Formatted at 2026-06-20 19:10:14 on dist-test-slave-wd30"
server_key: "72defdada34cf5447be9057b8b64371f"
server_key_iv: "57704a456139b2e3c10af0a74f9dcced"
server_key_version: "kuduclusterkey@0"
I20260620 19:10:16.584604 13515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:16.616990 13515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:16.619423 13515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:16.619590 13515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:16.624385 13515 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:44563
I20260620 19:10:16.624482 13583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:44563 every 8 connection(s)
I20260620 19:10:16.624908 13515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:16.626288 13584 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:10:16.627038 13584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702: Bootstrap starting.
I20260620 19:10:16.634744 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 13515
I20260620 19:10:16.635177 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.20:37927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:42131
--local_ip_for_outbound_sockets=127.0.203.1
--tserver_master_addrs=127.0.203.62:44563
--webserver_port=41623
--webserver_interface=127.0.203.1
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.203.20:37927/kms with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:10:16.635752 13584 log.cc:824] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:16.637769 13584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:10:16.638067 13584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702: Bootstrap complete.
I20260620 19:10:16.640798 13584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "954ab55185a24a0d85e3a95878dd3702" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44563 } }
I20260620 19:10:16.641070 13584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954ab55185a24a0d85e3a95878dd3702, State: Initialized, Role: FOLLOWER
I20260620 19:10:16.641281 13584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "954ab55185a24a0d85e3a95878dd3702" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44563 } }
I20260620 19:10:16.641379 13584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:16.641433 13584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:16.641486 13584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:10:16.642532 13584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "954ab55185a24a0d85e3a95878dd3702" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44563 } }
I20260620 19:10:16.642659 13584 leader_election.cc:302] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 954ab55185a24a0d85e3a95878dd3702; no voters:
I20260620 19:10:16.642856 13584 leader_election.cc:288] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:10:16.642899 13588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:10:16.643016 13588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [term 2 LEADER]: Becoming Leader. State: Replica: 954ab55185a24a0d85e3a95878dd3702, State: Running, Role: LEADER
I20260620 19:10:16.643152 13584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:16.643261 13588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "954ab55185a24a0d85e3a95878dd3702" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44563 } }
I20260620 19:10:16.643965 13589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 954ab55185a24a0d85e3a95878dd3702. Latest consensus state: current_term: 2 leader_uuid: "954ab55185a24a0d85e3a95878dd3702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "954ab55185a24a0d85e3a95878dd3702" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44563 } } }
I20260620 19:10:16.644047 13589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:16.644176 13590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "954ab55185a24a0d85e3a95878dd3702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "954ab55185a24a0d85e3a95878dd3702" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44563 } } }
I20260620 19:10:16.644243 13590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:16.644402 13598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:16.644972 13598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:16.645151 13598 catalog_manager.cc:1259] Loaded cluster ID: 858db870de3e4a1283fb892f18dc2732
I20260620 19:10:16.645224 13598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:16.645391 13584 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 19:10:16.648928 13598 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:16.649576 13598 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702: Loaded TSK: 0
I20260620 19:10:16.649910 13598 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:10:16.753875 13587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:16.754278 13587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:16.754396 13587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:16.756345 13587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:16.756412 13587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:16.756455 13587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:10:16.758755 13587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.203.20:37927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:42131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=41623
--tserver_master_addrs=127.0.203.62:44563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:16.759295 13587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:16.759677 13587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:16.766453 13622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:16.767234 13621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:16.767244 13624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:16.769335 13587 server_base.cc:1034] running on GCE node
I20260620 19:10:16.769958 13587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:16.770316 13587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:16.771499 13587 hybrid_clock.cc:648] HybridClock initialized: now 1781982616771466 us; error 53 us; skew 500 ppm
Jun 20 19:10:16 dist-test-slave-wd30 krb5kdc[12552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:16.774914 13587 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:10:16.775478 13587 webserver.cc:460] Webserver started at http://127.0.203.1:41623/ using document root <none> and password file <none>
I20260620 19:10:16.775750 13587 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:16.775820 13587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:10:16 dist-test-slave-wd30 krb5kdc[12552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
I20260620 19:10:16.796571 13587 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:16.797969 13632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:16.798264 13587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:16.798342 13587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "d3a9894a527c4fdb958d8f71c6903f2c"
format_stamp: "Formatted at 2026-06-20 19:10:15 on dist-test-slave-wd30"
server_key: "1d59a094398540340fbcbb05b11429d4"
server_key_iv: "e8d449054c54dd18f680c2d141b05858"
server_key_version: "kuduclusterkey@0"
I20260620 19:10:16.798834 13587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:16.814363 13587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:16.816782 13587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:16.816939 13587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:16.817158 13587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:16.817516 13587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:16.817564 13587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:16.817595 13587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:16.817620 13587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:16.823770 13587 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:42131
I20260620 19:10:16.824177 13587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:10:16.826822 13745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:42131 every 8 connection(s)
I20260620 19:10:16.833652 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 13587
I20260620 19:10:16.833941 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.20:37927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:45867
--local_ip_for_outbound_sockets=127.0.203.2
--tserver_master_addrs=127.0.203.62:44563
--webserver_port=44601
--webserver_interface=127.0.203.2
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.203.20:37927/kms with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:10:16 dist-test-slave-wd30 krb5kdc[12552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:16.849906 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:16.827429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:44563 (local address 127.0.203.1:51853)
0620 19:10:16.827745 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:16.827787 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:16.828369 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:16.829068 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:16.829072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:16.829278 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:16.829563 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:16.829568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.831261 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.831262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:16.832574 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:16.832577 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.835382 (+ 2805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.835384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:16.835552 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:16.843511 (+ 7959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:16.843522 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:16.845719 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:16.847926 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:16.847933 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:16.847935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:16.848217 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:16.848505 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:16.848506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:16.848507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:16.848653 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:16.849171 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:16.849177 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:16.849335 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 19:10:16.849442 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 19:10:16.849622 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":91,"threads_started":1}
I20260620 19:10:16.850553 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:16.827554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:51853 (local address 127.0.203.62:44563)
0620 19:10:16.828584 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:16.828592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:16.828627 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:10:16.828764 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:16.828769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:16.828819 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:16.828931 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:16.829693 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.831103 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.832707 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.833295 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.833398 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:16.845869 (+ 12471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:16.845874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:16.845874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:16.845886 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:16.847829 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:16.848325 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:16.848328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:16.848328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:16.848430 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:16.848736 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:16.848737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:16.848738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:16.849059 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:16.849140 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:16.850201 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0620 19:10:16.850327 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":975,"thread_start_us":68,"threads_started":1}
I20260620 19:10:16.851217 13746 heartbeater.cc:347] Connected to a master server at 127.0.203.62:44563
I20260620 19:10:16.851341 13746 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:16.851532 13746 heartbeater.cc:511] Master 127.0.203.62:44563 requested a full tablet report, sending...
W20260620 19:10:16.852427 13746 heartbeater.cc:644] Failed to heartbeat to 127.0.203.62:44563 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 19:10:16.852521 13746 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:16.852591 13746 heartbeater.cc:511] Master 127.0.203.62:44563 requested a full tablet report, sending...
I20260620 19:10:16.852913 13746 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:16.852988 13746 heartbeater.cc:511] Master 127.0.203.62:44563 requested a full tablet report, sending...
W20260620 19:10:16.853281 13746 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:10:16.954957 13750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:16.955231 13750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:16.955273 13750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:16.956460 13750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:16.956525 13750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:16.956570 13750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:10:16.958005 13750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.203.20:37927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:45867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=44601
--tserver_master_addrs=127.0.203.62:44563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:16.958256 13750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:16.958509 13750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:16.963845 13758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:16.964144 13755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:16.964344 13756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:16.965229 13750 server_base.cc:1034] running on GCE node
I20260620 19:10:16.965440 13750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:16.965679 13750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:16.966854 13750 hybrid_clock.cc:648] HybridClock initialized: now 1781982616966833 us; error 33 us; skew 500 ppm
Jun 20 19:10:16 dist-test-slave-wd30 krb5kdc[12552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:16.970064 13750 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:10:16.970577 13750 webserver.cc:460] Webserver started at http://127.0.203.2:44601/ using document root <none> and password file <none>
I20260620 19:10:16.970830 13750 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:16.970919 13750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:10:16 dist-test-slave-wd30 krb5kdc[12552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
I20260620 19:10:16.985405 13750 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:10:16.986357 13766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:16.986526 13750 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 19:10:16.986613 13750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "d3cc2a7f85274ddb99f1fe8507f28051"
format_stamp: "Formatted at 2026-06-20 19:10:16 on dist-test-slave-wd30"
server_key: "02a4912ef5d6997cb0ff38984519c1fd"
server_key_iv: "cde4ed098a4d15883dea50084b134070"
server_key_version: "kuduclusterkey@0"
I20260620 19:10:16.987004 13750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:17.008416 13750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:17.010632 13750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:17.010818 13750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:17.011088 13750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:17.011468 13750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:17.011528 13750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:17.011567 13750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:17.011591 13750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:17.018920 13750 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:45867
I20260620 19:10:17.019140 13879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:45867 every 8 connection(s)
I20260620 19:10:17.019414 13750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:10:17.024943 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 13750
I20260620 19:10:17.025252 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.20:37927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:37513
--local_ip_for_outbound_sockets=127.0.203.3
--tserver_master_addrs=127.0.203.62:44563
--webserver_port=44651
--webserver_interface=127.0.203.3
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.203.20:37927/kms with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:10:17 dist-test-slave-wd30 krb5kdc[12552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:17.032760 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:17.020151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:44563 (local address 127.0.203.2:38483)
0620 19:10:17.020357 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:17.020406 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:17.021054 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:17.021308 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:17.021312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:17.021541 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:17.021855 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:17.021862 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:17.022730 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:17.022731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:17.024074 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:17.024078 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:17.024603 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:17.024605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:17.026714 (+ 2109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:17.027362 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:17.027372 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:17.028970 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:17.030996 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:17.031003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:17.031005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:17.031385 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:17.031655 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:17.031656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:17.031656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:17.031787 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:17.032103 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:17.032110 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:17.032267 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 19:10:17.032362 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 19:10:17.032515 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":110,"threads_started":1}
I20260620 19:10:17.033478 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:17.020171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:38483 (local address 127.0.203.62:44563)
0620 19:10:17.020250 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:17.020252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:17.020516 (+ 264us) server_negotiation.cc:407] Connection header received
0620 19:10:17.021166 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:17.021167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:17.021179 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:17.021223 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:17.021943 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:17.022616 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:17.024168 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:17.024484 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:17.024561 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:17.029083 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:17.029085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:17.029085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:17.029093 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:17.030921 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:17.031509 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:17.031510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:17.031511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:17.031574 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:17.031870 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:17.031871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:17.031871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:17.032001 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:17.032060 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:17.033294 (+ 1234us) server_negotiation.cc:299] Negotiation successful
0620 19:10:17.033386 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:10:17.034046 13880 heartbeater.cc:347] Connected to a master server at 127.0.203.62:44563
I20260620 19:10:17.034190 13880 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:17.034415 13880 heartbeater.cc:511] Master 127.0.203.62:44563 requested a full tablet report, sending...
W20260620 19:10:17.035034 13880 heartbeater.cc:644] Failed to heartbeat to 127.0.203.62:44563 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 19:10:17.035189 13880 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:17.035288 13880 heartbeater.cc:511] Master 127.0.203.62:44563 requested a full tablet report, sending...
I20260620 19:10:17.035588 13880 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:17.035699 13880 heartbeater.cc:511] Master 127.0.203.62:44563 requested a full tablet report, sending...
W20260620 19:10:17.036019 13880 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:10:17.147718 13883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:17.147976 13883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:17.148012 13883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:17.149690 13883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:17.149735 13883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:17.149775 13883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:10:17.151993 13883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.203.20:37927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:37513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=44651
--tserver_master_addrs=127.0.203.62:44563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:17.152241 13883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:17.152500 13883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:17.157836 13890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:17.157845 13889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:17.157856 13892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:17.158033 13883 server_base.cc:1034] running on GCE node
I20260620 19:10:17.158262 13883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:17.158509 13883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:17.159633 13883 hybrid_clock.cc:648] HybridClock initialized: now 1781982617159608 us; error 39 us; skew 500 ppm
Jun 20 19:10:17 dist-test-slave-wd30 krb5kdc[12552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:17.162623 13883 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:10:17.163211 13883 webserver.cc:460] Webserver started at http://127.0.203.3:44651/ using document root <none> and password file <none>
I20260620 19:10:17.163509 13883 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:17.163591 13883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:10:17 dist-test-slave-wd30 krb5kdc[12552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
I20260620 19:10:17.180764 13883 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260620 19:10:17.183207 13900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:17.183460 13883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 19:10:17.183604 13883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "648908d936904f88a368ebf753262fec"
format_stamp: "Formatted at 2026-06-20 19:10:16 on dist-test-slave-wd30"
server_key: "0fcd65b69d21be609f19da2e2995e2c6"
server_key_iv: "c9fa89ff34d3a1d067920ed8adf26ed9"
server_key_version: "kuduclusterkey@0"
I20260620 19:10:17.184161 13883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:17.195815 13883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:17.197916 13883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:17.198073 13883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:17.198331 13883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:17.198709 13883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:17.198762 13883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:17.198796 13883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:17.198822 13883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:17.207451 13883 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:37513
I20260620 19:10:17.207664 14013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:37513 every 8 connection(s)
I20260620 19:10:17.208046 13883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:17.213799 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 13883
Jun 20 19:10:17 dist-test-slave-wd30 krb5kdc[12552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:17.219959 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:17.209367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:53349 (local address 127.0.203.62:44563)
0620 19:10:17.209412 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:17.209414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:17.209426 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:10:17.210079 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:17.210081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:17.210094 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:17.210165 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:17.210873 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:17.211700 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:17.213196 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:17.213522 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:17.213564 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:17.216604 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:17.216606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:17.216606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:17.216614 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:17.218229 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:17.218747 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:17.218748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:17.218749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:17.218813 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:17.219197 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:17.219198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:17.219198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:17.219346 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:17.219422 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:17.219777 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 19:10:17.219859 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:10:17.220165 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:17.209069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:44563 (local address 127.0.203.3:53349)
0620 19:10:17.209269 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:17.209317 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:17.209949 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:17.210237 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:17.210242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:17.210469 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:17.210762 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:17.210769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:17.211821 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:17.211822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:17.213092 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:17.213095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:17.214135 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:17.214136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:17.214433 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:17.215017 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:17.215027 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:17.216497 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:17.218328 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:17.218337 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:17.218339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:17.218650 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:17.218918 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:17.218919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:17.218920 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:17.219064 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:17.219432 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:17.219439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:17.219616 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 19:10:17.219739 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 19:10:17.219898 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":100,"threads_started":1}
I20260620 19:10:17.220746 14014 heartbeater.cc:347] Connected to a master server at 127.0.203.62:44563
I20260620 19:10:17.220858 14014 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:17.221056 14014 heartbeater.cc:511] Master 127.0.203.62:44563 requested a full tablet report, sending...
W20260620 19:10:17.221803 14014 heartbeater.cc:644] Failed to heartbeat to 127.0.203.62:44563 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 19:10:17.221881 14014 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:17.221949 14014 heartbeater.cc:511] Master 127.0.203.62:44563 requested a full tablet report, sending...
I20260620 19:10:17.222522 14014 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:17.222606 14014 heartbeater.cc:511] Master 127.0.203.62:44563 requested a full tablet report, sending...
W20260620 19:10:17.222908 14014 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:10:17.223400 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:17.214218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44652 (local address 127.0.203.62:44563)
0620 19:10:17.215231 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:17.215233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:17.215411 (+ 178us) server_negotiation.cc:407] Connection header received
0620 19:10:17.215457 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:17.215458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:17.215474 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:17.215549 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:17.216295 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:17.216938 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:17.218027 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:17.218361 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:17.218446 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:17.219820 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:17.219822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:17.219822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:17.219829 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:17.221902 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:17.222256 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:17.222257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:17.222257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:17.222329 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:17.222555 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:17.222556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:17.222557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:17.222690 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:17.222856 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:17.223211 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 19:10:17.223297 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":993,"thread_start_us":63,"threads_started":1}
Jun 20 19:10:17 dist-test-slave-wd30 krb5kdc[12552](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:17.853480 13746 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:17.853660 13746 heartbeater.cc:511] Master 127.0.203.62:44563 requested a full tablet report, sending...
I20260620 19:10:17.854132 13548 ts_manager.cc:194] Registered new tserver with Master: d3a9894a527c4fdb958d8f71c6903f2c (127.0.203.1:42131)
I20260620 19:10:17.855171 13548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:51853
I20260620 19:10:18.036273 13880 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:18.036454 13880 heartbeater.cc:511] Master 127.0.203.62:44563 requested a full tablet report, sending...
I20260620 19:10:18.036844 13548 ts_manager.cc:194] Registered new tserver with Master: d3cc2a7f85274ddb99f1fe8507f28051 (127.0.203.2:45867)
I20260620 19:10:18.037403 13548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:38483
Jun 20 19:10:18 dist-test-slave-wd30 krb5kdc[12552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
I20260620 19:10:18.223100 14014 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:18.223292 14014 heartbeater.cc:511] Master 127.0.203.62:44563 requested a full tablet report, sending...
I20260620 19:10:18.223579 13548 ts_manager.cc:194] Registered new tserver with Master: 648908d936904f88a368ebf753262fec (127.0.203.3:37513)
I20260620 19:10:18.224120 13548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:53349
I20260620 19:10:18.229140 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:10:18.238121 14036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.230124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44664 (local address 127.0.203.62:44563)
0620 19:10:18.230363 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:18.230365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:18.230372 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:10:18.230395 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:18.230395 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:18.230407 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:18.230481 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:18.231212 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.231979 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.232966 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.233276 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.233354 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.234800 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:18.234802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:18.234802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:18.234809 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:18.236681 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:18.237083 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:18.237085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:18.237085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:18.237154 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:18.237375 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:18.237376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:18.237376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:18.237523 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:18.237643 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:18.237796 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 19:10:18.237886 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":64,"threads_started":1}
I20260620 19:10:18.240789 13548 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:10:18.243427 13548 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:18.255517 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.246716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:42131 (local address 127.0.0.1:45882)
0620 19:10:18.247797 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:18.247811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:18.247876 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:18.248503 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:18.248504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:18.248506 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:18.248552 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.248554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.249796 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.249797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.253449 (+ 3652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.253452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.255208 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.255209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.255297 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.255311 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 19:10:18.255378 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:10:18.255426 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"thread_start_us":96,"threads_started":1}
I20260620 19:10:18.255930 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.247623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.0.203.3:37513)
0620 19:10:18.247789 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:18.247794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:18.251636 (+ 3842us) server_negotiation.cc:407] Connection header received
0620 19:10:18.251676 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:18.251679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:18.251694 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:18.251776 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:18.251997 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.253038 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.254602 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.255346 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.255415 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.255464 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:18.255812 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 19:10:18.255853 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260620 19:10:18.256287 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.247202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41902 (local address 127.0.203.2:45867)
0620 19:10:18.248006 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:18.248010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:18.248020 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:10:18.248057 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:18.248059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:18.248074 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:18.248156 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:18.248356 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.249808 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.253538 (+ 3730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.254240 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.256071 (+ 1831us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.256135 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:18.256160 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 19:10:18.256202 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":77,"threads_started":1}
I20260620 19:10:18.255719 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.246949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:37513 (local address 127.0.0.1:52672)
0620 19:10:18.251244 (+ 4295us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:18.251255 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:18.251312 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:18.251869 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:18.251870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:18.251873 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:18.251913 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.251914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.253135 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.253136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.254518 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.254520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.255498 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.255499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.255558 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.255562 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:10:18.255611 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:10:18.255647 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4272,"thread_start_us":66,"threads_started":1}
I20260620 19:10:18.256009 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.246711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:45867 (local address 127.0.0.1:41902)
0620 19:10:18.247096 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:18.247162 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:18.247532 (+ 370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:18.248176 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:18.248179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:18.248181 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:18.248261 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.248263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.251233 (+ 2970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.251234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.252427 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.252430 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.255832 (+ 3402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.255833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.255891 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.255894 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:10:18.255922 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 19:10:18.255955 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":106,"threads_started":1}
I20260620 19:10:18.256829 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.247462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45882 (local address 127.0.203.1:42131)
0620 19:10:18.248287 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:18.248291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:18.248300 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:10:18.248332 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:18.248334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:18.248348 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:18.248418 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:18.248631 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.249698 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.254333 (+ 4635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.255020 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.255087 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.255174 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:18.256702 (+ 1528us) server_negotiation.cc:299] Negotiation successful
0620 19:10:18.256753 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":64,"threads_started":1}
I20260620 19:10:18.257329 13814 tablet_service.cc:1467] Processing CreateTablet for tablet 44ee0dccc21242ffb6dd12f878992adb (DEFAULT_TABLE table=test-table [id=5d5de176a97c42d09737d6942adf27d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:18.257328 13929 tablet_service.cc:1467] Processing CreateTablet for tablet 44ee0dccc21242ffb6dd12f878992adb (DEFAULT_TABLE table=test-table [id=5d5de176a97c42d09737d6942adf27d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:18.257689 13814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44ee0dccc21242ffb6dd12f878992adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:18.257689 13929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44ee0dccc21242ffb6dd12f878992adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:18.259687 13680 tablet_service.cc:1467] Processing CreateTablet for tablet 44ee0dccc21242ffb6dd12f878992adb (DEFAULT_TABLE table=test-table [id=5d5de176a97c42d09737d6942adf27d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:18.260006 13680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44ee0dccc21242ffb6dd12f878992adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:18.266182 14049 tablet_bootstrap.cc:492] T 44ee0dccc21242ffb6dd12f878992adb P d3a9894a527c4fdb958d8f71c6903f2c: Bootstrap starting.
I20260620 19:10:18.266474 14047 tablet_bootstrap.cc:492] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051: Bootstrap starting.
I20260620 19:10:18.267020 14049 tablet_bootstrap.cc:654] T 44ee0dccc21242ffb6dd12f878992adb P d3a9894a527c4fdb958d8f71c6903f2c: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:18.267107 14047 tablet_bootstrap.cc:654] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:18.267465 14047 log.cc:824] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:18.267486 14049 log.cc:824] T 44ee0dccc21242ffb6dd12f878992adb P d3a9894a527c4fdb958d8f71c6903f2c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:18.267814 14048 tablet_bootstrap.cc:492] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec: Bootstrap starting.
I20260620 19:10:18.268455 14048 tablet_bootstrap.cc:654] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:18.268718 14047 tablet_bootstrap.cc:492] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051: No bootstrap required, opened a new log
I20260620 19:10:18.268803 14048 log.cc:824] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:18.268852 14047 ts_tablet_manager.cc:1397] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:18.269418 14049 tablet_bootstrap.cc:492] T 44ee0dccc21242ffb6dd12f878992adb P d3a9894a527c4fdb958d8f71c6903f2c: No bootstrap required, opened a new log
I20260620 19:10:18.269490 14049 ts_tablet_manager.cc:1397] T 44ee0dccc21242ffb6dd12f878992adb P d3a9894a527c4fdb958d8f71c6903f2c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 19:10:18.272253 14047 raft_consensus.cc:348] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9894a527c4fdb958d8f71c6903f2c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42131 } } peers { permanent_uuid: "d3cc2a7f85274ddb99f1fe8507f28051" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45867 } } peers { permanent_uuid: "648908d936904f88a368ebf753262fec" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37513 } }
I20260620 19:10:18.272491 14049 raft_consensus.cc:348] T 44ee0dccc21242ffb6dd12f878992adb P d3a9894a527c4fdb958d8f71c6903f2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9894a527c4fdb958d8f71c6903f2c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42131 } } peers { permanent_uuid: "d3cc2a7f85274ddb99f1fe8507f28051" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45867 } } peers { permanent_uuid: "648908d936904f88a368ebf753262fec" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37513 } }
I20260620 19:10:18.272598 14047 raft_consensus.cc:374] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:18.272650 14047 raft_consensus.cc:729] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cc2a7f85274ddb99f1fe8507f28051, State: Initialized, Role: FOLLOWER
I20260620 19:10:18.272771 14049 raft_consensus.cc:374] T 44ee0dccc21242ffb6dd12f878992adb P d3a9894a527c4fdb958d8f71c6903f2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:18.272802 14047 consensus_queue.cc:260] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9894a527c4fdb958d8f71c6903f2c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42131 } } peers { permanent_uuid: "d3cc2a7f85274ddb99f1fe8507f28051" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45867 } } peers { permanent_uuid: "648908d936904f88a368ebf753262fec" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37513 } }
I20260620 19:10:18.272876 14049 raft_consensus.cc:729] T 44ee0dccc21242ffb6dd12f878992adb P d3a9894a527c4fdb958d8f71c6903f2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a9894a527c4fdb958d8f71c6903f2c, State: Initialized, Role: FOLLOWER
I20260620 19:10:18.273101 14047 ts_tablet_manager.cc:1428] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 19:10:18.273100 14049 consensus_queue.cc:260] T 44ee0dccc21242ffb6dd12f878992adb P d3a9894a527c4fdb958d8f71c6903f2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9894a527c4fdb958d8f71c6903f2c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42131 } } peers { permanent_uuid: "d3cc2a7f85274ddb99f1fe8507f28051" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45867 } } peers { permanent_uuid: "648908d936904f88a368ebf753262fec" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37513 } }
I20260620 19:10:18.273280 13880 heartbeater.cc:503] Master 127.0.203.62:44563 was elected leader, sending a full tablet report...
I20260620 19:10:18.273435 14049 ts_tablet_manager.cc:1428] T 44ee0dccc21242ffb6dd12f878992adb P d3a9894a527c4fdb958d8f71c6903f2c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 19:10:18.273537 13746 heartbeater.cc:503] Master 127.0.203.62:44563 was elected leader, sending a full tablet report...
I20260620 19:10:18.273891 14048 tablet_bootstrap.cc:492] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec: No bootstrap required, opened a new log
I20260620 19:10:18.273953 14048 ts_tablet_manager.cc:1397] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 19:10:18.276528 14048 raft_consensus.cc:348] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9894a527c4fdb958d8f71c6903f2c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42131 } } peers { permanent_uuid: "d3cc2a7f85274ddb99f1fe8507f28051" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45867 } } peers { permanent_uuid: "648908d936904f88a368ebf753262fec" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37513 } }
I20260620 19:10:18.276698 14048 raft_consensus.cc:374] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:18.276731 14048 raft_consensus.cc:729] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648908d936904f88a368ebf753262fec, State: Initialized, Role: FOLLOWER
I20260620 19:10:18.276834 14048 consensus_queue.cc:260] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9894a527c4fdb958d8f71c6903f2c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42131 } } peers { permanent_uuid: "d3cc2a7f85274ddb99f1fe8507f28051" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45867 } } peers { permanent_uuid: "648908d936904f88a368ebf753262fec" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37513 } }
I20260620 19:10:18.277107 14048 ts_tablet_manager.cc:1428] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 19:10:18.277124 14014 heartbeater.cc:503] Master 127.0.203.62:44563 was elected leader, sending a full tablet report...
W20260620 19:10:18.328189 13747 tablet.cc:2369] T 44ee0dccc21242ffb6dd12f878992adb P d3a9894a527c4fdb958d8f71c6903f2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 19:10:18 dist-test-slave-wd30 krb5kdc[12552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for HTTP/127.0.203.20@KRBTEST.COM
I20260620 19:10:18.383482 14055 raft_consensus.cc:482] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:18.383647 14055 raft_consensus.cc:504] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9894a527c4fdb958d8f71c6903f2c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42131 } } peers { permanent_uuid: "d3cc2a7f85274ddb99f1fe8507f28051" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45867 } } peers { permanent_uuid: "648908d936904f88a368ebf753262fec" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37513 } }
I20260620 19:10:18.384040 14055 leader_election.cc:288] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a9894a527c4fdb958d8f71c6903f2c (127.0.203.1:42131), d3cc2a7f85274ddb99f1fe8507f28051 (127.0.203.2:45867)
I20260620 19:10:18.391152 14054 raft_consensus.cc:482] T 44ee0dccc21242ffb6dd12f878992adb P d3a9894a527c4fdb958d8f71c6903f2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:18.391292 14054 raft_consensus.cc:504] T 44ee0dccc21242ffb6dd12f878992adb P d3a9894a527c4fdb958d8f71c6903f2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9894a527c4fdb958d8f71c6903f2c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42131 } } peers { permanent_uuid: "d3cc2a7f85274ddb99f1fe8507f28051" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45867 } } peers { permanent_uuid: "648908d936904f88a368ebf753262fec" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37513 } }
I20260620 19:10:18.391423 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.384600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:52851 (local address 127.0.203.2:45867)
0620 19:10:18.384708 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:18.384710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:18.384855 (+ 145us) server_negotiation.cc:407] Connection header received
0620 19:10:18.385892 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:18.385894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:18.385906 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:18.385934 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:18.386124 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.386840 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.390299 (+ 3459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.390902 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.390949 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.390959 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:18.391304 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 19:10:18.391352 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 19:10:18.391690 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.384227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:45867 (local address 127.0.203.3:52851)
0620 19:10:18.384796 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:18.384806 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:18.385800 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:18.385995 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:18.385995 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:18.385997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:18.386058 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.386059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.386936 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.386937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.390206 (+ 3269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.390209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.391149 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.391150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.391215 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.391219 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:10:18.391541 (+ 322us) client_negotiation.cc:240] Negotiation successful
0620 19:10:18.391585 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":100,"threads_started":1}
I20260620 19:10:18.391712 14054 leader_election.cc:288] T 44ee0dccc21242ffb6dd12f878992adb P d3a9894a527c4fdb958d8f71c6903f2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cc2a7f85274ddb99f1fe8507f28051 (127.0.203.2:45867), 648908d936904f88a368ebf753262fec (127.0.203.3:37513)
I20260620 19:10:18.391997 13821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ee0dccc21242ffb6dd12f878992adb" candidate_uuid: "648908d936904f88a368ebf753262fec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cc2a7f85274ddb99f1fe8507f28051" is_pre_election: true
I20260620 19:10:18.392148 13821 raft_consensus.cc:2454] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 648908d936904f88a368ebf753262fec in term 0.
I20260620 19:10:18.392423 13902 leader_election.cc:302] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 648908d936904f88a368ebf753262fec, d3cc2a7f85274ddb99f1fe8507f28051; no voters:
I20260620 19:10:18.392544 14055 raft_consensus.cc:2781] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:18.392588 14055 raft_consensus.cc:482] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:18.392616 14055 raft_consensus.cc:3037] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:18.393453 14055 raft_consensus.cc:504] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9894a527c4fdb958d8f71c6903f2c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42131 } } peers { permanent_uuid: "d3cc2a7f85274ddb99f1fe8507f28051" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45867 } } peers { permanent_uuid: "648908d936904f88a368ebf753262fec" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37513 } }
I20260620 19:10:18.393574 14055 leader_election.cc:288] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [CANDIDATE]: Term 1 election: Requested vote from peers d3a9894a527c4fdb958d8f71c6903f2c (127.0.203.1:42131), d3cc2a7f85274ddb99f1fe8507f28051 (127.0.203.2:45867)
I20260620 19:10:18.393841 13821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ee0dccc21242ffb6dd12f878992adb" candidate_uuid: "648908d936904f88a368ebf753262fec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cc2a7f85274ddb99f1fe8507f28051"
I20260620 19:10:18.393911 13821 raft_consensus.cc:3037] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:18.394685 13821 raft_consensus.cc:2454] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648908d936904f88a368ebf753262fec in term 1.
I20260620 19:10:18.394902 13902 leader_election.cc:302] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 648908d936904f88a368ebf753262fec, d3cc2a7f85274ddb99f1fe8507f28051; no voters:
I20260620 19:10:18.395046 14055 raft_consensus.cc:2781] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:18.395188 14055 raft_consensus.cc:686] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [term 1 LEADER]: Becoming Leader. State: Replica: 648908d936904f88a368ebf753262fec, State: Running, Role: LEADER
I20260620 19:10:18.395273 14055 consensus_queue.cc:237] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9894a527c4fdb958d8f71c6903f2c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42131 } } peers { permanent_uuid: "d3cc2a7f85274ddb99f1fe8507f28051" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45867 } } peers { permanent_uuid: "648908d936904f88a368ebf753262fec" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37513 } }
I20260620 19:10:18.396119 13547 catalog_manager.cc:5495] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec reported cstate change: term changed from 0 to 1, leader changed from <none> to 648908d936904f88a368ebf753262fec (127.0.203.3). New cstate: current_term: 1 leader_uuid: "648908d936904f88a368ebf753262fec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a9894a527c4fdb958d8f71c6903f2c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3cc2a7f85274ddb99f1fe8507f28051" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "648908d936904f88a368ebf753262fec" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37513 } health_report { overall_health: HEALTHY } } }
I20260620 19:10:18.400588 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.392273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:51415 (local address 127.0.203.2:45867)
0620 19:10:18.395021 (+ 2748us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:18.395023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:18.396379 (+ 1356us) server_negotiation.cc:407] Connection header received
0620 19:10:18.397244 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:18.397245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:18.397255 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:18.397283 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:18.397542 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.398229 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.399706 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.400225 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.400269 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.400278 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:18.400493 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 19:10:18.400526 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2726}
I20260620 19:10:18.400842 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.392085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:45867 (local address 127.0.203.1:51415)
0620 19:10:18.396299 (+ 4214us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:18.396314 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:18.396447 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:18.397385 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:18.397386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:18.397389 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:18.397469 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.397471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.398328 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.398329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.399608 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.399612 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.400348 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.400349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.400418 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.400422 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:10:18.400708 (+ 286us) client_negotiation.cc:240] Negotiation successful
0620 19:10:18.400754 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4190,"thread_start_us":97,"threads_started":1}
I20260620 19:10:18.401269 13821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ee0dccc21242ffb6dd12f878992adb" candidate_uuid: "d3a9894a527c4fdb958d8f71c6903f2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cc2a7f85274ddb99f1fe8507f28051" is_pre_election: true
I20260620 19:10:18.401363 13821 raft_consensus.cc:2379] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3a9894a527c4fdb958d8f71c6903f2c in current term 1: Already voted for candidate 648908d936904f88a368ebf753262fec in this term.
I20260620 19:10:18.405174 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.384662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:40561 (local address 127.0.203.1:42131)
0620 19:10:18.384740 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:18.384742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:18.384992 (+ 250us) server_negotiation.cc:407] Connection header received
0620 19:10:18.385101 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:18.385103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:18.385117 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:18.385149 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:18.385400 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.386121 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.389669 (+ 3548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.390262 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.390310 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.390321 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:18.405060 (+ 14739us) server_negotiation.cc:299] Negotiation successful
0620 19:10:18.405112 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 19:10:18.405184 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.384289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:42131 (local address 127.0.203.3:40561)
0620 19:10:18.384928 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:18.384939 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:18.385023 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:18.385237 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:18.385238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:18.385239 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:18.385319 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.385320 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.387092 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.387093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.389569 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.389573 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.404890 (+ 15317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.404891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.404972 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.404977 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:10:18.405039 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:10:18.405081 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":94,"threads_started":1}
I20260620 19:10:18.405527 13700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ee0dccc21242ffb6dd12f878992adb" candidate_uuid: "648908d936904f88a368ebf753262fec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a9894a527c4fdb958d8f71c6903f2c" is_pre_election: true
I20260620 19:10:18.405644 13700 raft_consensus.cc:2454] T 44ee0dccc21242ffb6dd12f878992adb P d3a9894a527c4fdb958d8f71c6903f2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 648908d936904f88a368ebf753262fec in term 0.
I20260620 19:10:18.405778 13699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ee0dccc21242ffb6dd12f878992adb" candidate_uuid: "648908d936904f88a368ebf753262fec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a9894a527c4fdb958d8f71c6903f2c"
I20260620 19:10:18.405846 13699 raft_consensus.cc:3037] T 44ee0dccc21242ffb6dd12f878992adb P d3a9894a527c4fdb958d8f71c6903f2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:18.406737 13699 raft_consensus.cc:2454] T 44ee0dccc21242ffb6dd12f878992adb P d3a9894a527c4fdb958d8f71c6903f2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648908d936904f88a368ebf753262fec in term 1.
I20260620 19:10:18.407243 14061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.392314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:37513 (local address 127.0.203.1:50405)
0620 19:10:18.400872 (+ 8558us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:18.400886 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:18.400970 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:18.401218 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:18.401218 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:18.401221 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:18.401283 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.401285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.402258 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.402259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.403437 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.403440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.406962 (+ 3522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.406963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.407033 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.407037 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:10:18.407094 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 19:10:18.407147 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8525,"thread_start_us":87,"threads_started":1}
I20260620 19:10:18.407251 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.392700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:50405 (local address 127.0.203.3:37513)
0620 19:10:18.392741 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:18.392743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:18.400942 (+ 8199us) server_negotiation.cc:407] Connection header received
0620 19:10:18.401041 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:18.401042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:18.401053 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:18.401080 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:18.401358 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.402062 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.403525 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.404143 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.404229 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.404240 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:18.407113 (+ 2873us) server_negotiation.cc:299] Negotiation successful
0620 19:10:18.407181 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:10:18.407661 13965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ee0dccc21242ffb6dd12f878992adb" candidate_uuid: "d3a9894a527c4fdb958d8f71c6903f2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648908d936904f88a368ebf753262fec" is_pre_election: true
I20260620 19:10:18.407984 13636 leader_election.cc:302] T 44ee0dccc21242ffb6dd12f878992adb P d3a9894a527c4fdb958d8f71c6903f2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a9894a527c4fdb958d8f71c6903f2c; no voters: 648908d936904f88a368ebf753262fec, d3cc2a7f85274ddb99f1fe8507f28051
I20260620 19:10:18.408147 14054 raft_consensus.cc:2726] T 44ee0dccc21242ffb6dd12f878992adb P d3a9894a527c4fdb958d8f71c6903f2c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:10:18.416487 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.413303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52676 (local address 127.0.203.3:37513)
0620 19:10:18.413340 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:18.413341 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:18.413354 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:10:18.413433 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:18.413433 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:18.413453 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:18.413540 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:18.413679 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.414375 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.415470 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.415766 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.415820 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.415993 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:18.416121 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:18.416189 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:18.416399 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 19:10:18.416418 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:10:18.418501 13821 raft_consensus.cc:1261] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051 [term 1 FOLLOWER]: Refusing update from remote peer 648908d936904f88a368ebf753262fec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:18.418850 14062 consensus_queue.cc:1035] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cc2a7f85274ddb99f1fe8507f28051" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 45867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:18.421044 13699 raft_consensus.cc:1261] T 44ee0dccc21242ffb6dd12f878992adb P d3a9894a527c4fdb958d8f71c6903f2c [term 1 FOLLOWER]: Refusing update from remote peer 648908d936904f88a368ebf753262fec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:18.421339 14062 consensus_queue.cc:1035] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a9894a527c4fdb958d8f71c6903f2c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:18.425415 14065 mvcc.cc:205] Tried to move back new op lower bound from 7299000805039845376 to 7299000804947984384. Current Snapshot: MvccSnapshot[applied={T|T < 7299000805039845376}]
I20260620 19:10:18.426023 14066 mvcc.cc:205] Tried to move back new op lower bound from 7299000805039845376 to 7299000804947984384. Current Snapshot: MvccSnapshot[applied={T|T < 7299000805039845376}]
I20260620 19:10:18.430016 13547 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:18.430120 13547 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:18.431038 13547 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 954ab55185a24a0d85e3a95878dd3702: Sending DeleteTablet for 3 replicas of tablet 44ee0dccc21242ffb6dd12f878992adb
I20260620 19:10:18.431420 13814 tablet_service.cc:1514] Processing DeleteTablet for tablet 44ee0dccc21242ffb6dd12f878992adb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:18 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:41902
I20260620 19:10:18.431532 13680 tablet_service.cc:1514] Processing DeleteTablet for tablet 44ee0dccc21242ffb6dd12f878992adb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:18 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:45882
I20260620 19:10:18.431731 13929 tablet_service.cc:1514] Processing DeleteTablet for tablet 44ee0dccc21242ffb6dd12f878992adb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:18 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:52672
I20260620 19:10:18.432410 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 13587
I20260620 19:10:18.432621 14075 tablet_replica.cc:330] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec: stopping tablet replica
I20260620 19:10:18.432724 14075 raft_consensus.cc:2229] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:10:18.434039 14075 raft_consensus.cc:2258] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:18.435925 14075 ts_tablet_manager.cc:1905] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:18.437742 14075 ts_tablet_manager.cc:1918] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:18.437872 14075 log.cc:1196] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/44ee0dccc21242ffb6dd12f878992adb
I20260620 19:10:18.438232 14075 ts_tablet_manager.cc:1939] T 44ee0dccc21242ffb6dd12f878992adb P 648908d936904f88a368ebf753262fec: Deleting consensus metadata
I20260620 19:10:18.438709 13535 catalog_manager.cc:4841] TS 648908d936904f88a368ebf753262fec (127.0.203.3:37513): tablet 44ee0dccc21242ffb6dd12f878992adb (table test-table [id=5d5de176a97c42d09737d6942adf27d8]) successfully deleted
I20260620 19:10:18.451085 14073 tablet_replica.cc:330] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051: stopping tablet replica
I20260620 19:10:18.451303 14073 raft_consensus.cc:2229] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:18.451403 14073 raft_consensus.cc:2258] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:18.453203 14073 ts_tablet_manager.cc:1905] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:18.454993 14073 ts_tablet_manager.cc:1918] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260620 19:10:18.464962 13904 connection.cc:535] client connection to 127.0.203.1:42131 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 19:10:18.465108 13904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:10:18.465327 14073 log.cc:1196] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/44ee0dccc21242ffb6dd12f878992adb
W20260620 19:10:18.465358 13535 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.1:42131 (error 108)
I20260620 19:10:18.465674 14073 ts_tablet_manager.cc:1939] T 44ee0dccc21242ffb6dd12f878992adb P d3cc2a7f85274ddb99f1fe8507f28051: Deleting consensus metadata
I20260620 19:10:18.465845 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 13750
I20260620 19:10:18.475533 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.475387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:42131 (local address 127.0.203.3:49297)
0620 19:10:18.475435 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:18.475485 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.1:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 19:10:18.475807 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.475644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:42131 (local address 127.0.0.1:45886)
0620 19:10:18.475712 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:18.475768 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.1:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 19:10:18.475924 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.475805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:45867 (local address 127.0.0.1:41918)
0620 19:10:18.475886 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:18.475909 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.2:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 19:10:18.476058 13535 catalog_manager.cc:4584] TS d3a9894a527c4fdb958d8f71c6903f2c (127.0.203.1:42131): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 44ee0dccc21242ffb6dd12f878992adb: Network error: Client connection negotiation failed: client connection to 127.0.203.1:42131: connect: Connection refused (error 111)
I20260620 19:10:18.476109 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 13883
I20260620 19:10:18.486097 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 13515
I20260620 19:10:18.493552 812 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 19:10:18.840955 812 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 19:10:18.841060 812 mini_ranger.cc:66] Stopping Ranger...
I20260620 19:10:19.185554 812 mini_ranger.cc:68] Stopped Ranger
2026-06-20 19:10:19.185 UTC [12582] LOG: received smart shutdown request
2026-06-20 19:10:19.189 UTC [12582] LOG: background worker "logical replication launcher" (PID 12591) exited with exit code 1
2026-06-20 19:10:19.189 UTC [12586] LOG: shutting down
2026-06-20 19:10:19.355 UTC [12582] LOG: database system is shut down
2026-06-20T19:10:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48650 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:10:19 dist-test-slave-wd30 krb5kdc[14086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:10:19 dist-test-slave-wd30 krb5kdc[14086](info): set up 2 sockets
Jun 20 19:10:19 dist-test-slave-wd30 krb5kdc[14086](info): commencing operation
krb5kdc: starting...
W20260620 19:10:20.519609 812 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:10:20 dist-test-slave-wd30 krb5kdc[14086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:10:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:10:20.638283 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:36847
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:36847,127.0.203.61:34173,127.0.203.60:40385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:20.722613 14099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:20.722868 14099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:20.722908 14099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:20.724118 14099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:20.724195 14099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:20.724218 14099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:20.724248 14099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:20.724278 14099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:20.725749 14099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:36847,127.0.203.61:34173,127.0.203.60:40385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:36847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:20.726027 14099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:20.726253 14099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:20.730939 14105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:20.731032 14099 server_base.cc:1034] running on GCE node
W20260620 19:10:20.730940 14104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:20.730939 14107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:20.731491 14099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:20.731786 14099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:20.732956 14099 hybrid_clock.cc:648] HybridClock initialized: now 1781982620732951 us; error 38 us; skew 500 ppm
Jun 20 19:10:20 dist-test-slave-wd30 krb5kdc[14086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:20.739595 14099 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:10:20.740042 14099 webserver.cc:460] Webserver started at http://127.0.203.62:33007/ using document root <none> and password file <none>
I20260620 19:10:20.740267 14099 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:20.740336 14099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:20.740490 14099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:20.741580 14099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "54f2114d3ed844999fc36f2265101eff"
format_stamp: "Formatted at 2026-06-20 19:10:20 on dist-test-slave-wd30"
server_key: "f13738f4b10f71b8accb6a7e32f13be6"
server_key_iv: "08983c5b1a73145874ac3c4f4b6da4c6"
server_key_version: "encryptionkey@0"
I20260620 19:10:20.741948 14099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "54f2114d3ed844999fc36f2265101eff"
format_stamp: "Formatted at 2026-06-20 19:10:20 on dist-test-slave-wd30"
server_key: "f13738f4b10f71b8accb6a7e32f13be6"
server_key_iv: "08983c5b1a73145874ac3c4f4b6da4c6"
server_key_version: "encryptionkey@0"
I20260620 19:10:20.743415 14099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:10:20.744194 14114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:20.744386 14099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:20.744488 14099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "54f2114d3ed844999fc36f2265101eff"
format_stamp: "Formatted at 2026-06-20 19:10:20 on dist-test-slave-wd30"
server_key: "f13738f4b10f71b8accb6a7e32f13be6"
server_key_iv: "08983c5b1a73145874ac3c4f4b6da4c6"
server_key_version: "encryptionkey@0"
I20260620 19:10:20.744575 14099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:20.755800 14099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:20.757647 14099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:20.757802 14099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:20.761822 14099 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:36847
I20260620 19:10:20.761855 14166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:36847 every 8 connection(s)
I20260620 19:10:20.762202 14099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:20.763165 14167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:20.763587 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 14099
I20260620 19:10:20.763674 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:10:20.763868 812 external_mini_cluster.cc:1428] Setting key db1d12de9b255b9286e1405418db11cc
I20260620 19:10:20.764853 14167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.203.61@KRBTEST.COM; defaulting to no policy
Jun 20 19:10:20 dist-test-slave-wd30 krb5kdc[14086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.61@KRBTEST.COM" created.
I20260620 19:10:20.775640 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.765712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39164 (local address 127.0.203.62:36847)
0620 19:10:20.765875 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:20.765887 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:20.766012 (+ 125us) server_negotiation.cc:407] Connection header received
0620 19:10:20.766513 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:20.766516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:20.766537 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:20.766642 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:20.767174 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.768585 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.769439 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.769964 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.770099 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:20.772381 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:20.772388 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:20.772389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:20.772396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:20.773927 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:20.774417 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:20.774420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:20.774420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:20.774504 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:20.774735 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:20.774736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:20.774736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:20.774941 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:20.775047 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:20.775240 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 19:10:20.775369 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260620 19:10:20.775645 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.765680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36847 (local address 127.0.0.1:39164)
0620 19:10:20.765878 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:20.765918 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:20.766363 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:20.766655 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:20.766657 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:20.766908 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:20.767056 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:20.767059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.768688 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.768689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:20.769349 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:20.769351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.770069 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.770070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:20.770126 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:20.770656 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:20.770664 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:20.772285 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:20.774066 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:20.774073 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:20.774074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:20.774325 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:20.774571 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:20.774572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:20.774573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:20.774663 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:20.775026 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:20.775031 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:20.775091 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 19:10:20.775203 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 19:10:20.775436 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":93,"thread_start_us":87,"threads_started":1}
I20260620 19:10:20.776527 14167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:10:20.776873 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.776753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:34173 (local address 127.0.0.1:46812)
0620 19:10:20.776801 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:20.776844 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.61:34173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 19:10:20.777031 14115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.203.61:34173: connect: Connection refused (error 111)
I20260620 19:10:20.777433 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.777371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:34173 (local address 127.0.0.1:46822)
0620 19:10:20.777397 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:20.777419 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.61:34173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260620 19:10:20.777547 14167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.61:34173: Network error: Client connection negotiation failed: client connection to 127.0.203.61:34173: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.61.keytab.
Entry for principal kudu/127.0.203.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.61.keytab.
I20260620 19:10:20.790512 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.61:34173
--webserver_interface=127.0.203.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.61.keytab
--principal=kudu/127.0.203.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:36847,127.0.203.61:34173,127.0.203.60:40385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:10:20.826761 14167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } attempt: 1
I20260620 19:10:20.827306 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.827178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:34173 (local address 127.0.0.1:46834)
0620 19:10:20.827238 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:20.827277 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.61:34173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 19:10:20.827622 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.827533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:34173 (local address 127.0.0.1:46836)
0620 19:10:20.827576 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:20.827600 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.61:34173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 19:10:20.827700 14167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.61:34173: Network error: Client connection negotiation failed: client connection to 127.0.203.61:34173: connect: Connection refused (error 111)
W20260620 19:10:20.871250 14174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:20.871470 14174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:20.871495 14174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:20.872671 14174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:20.872716 14174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:20.872731 14174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:20.872745 14174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:20.872756 14174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:20.874261 14174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:36847,127.0.203.61:34173,127.0.203.60:40385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.61.keytab
--principal=kudu/127.0.203.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.61:34173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.203.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:20.874526 14174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:20.874765 14174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:20.879470 14179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:20.879562 14180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:20.879518 14182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:20.879652 14174 server_base.cc:1034] running on GCE node
I20260620 19:10:20.879928 14174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:20.880259 14174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:20.881434 14174 hybrid_clock.cc:648] HybridClock initialized: now 1781982620881420 us; error 49 us; skew 500 ppm
Jun 20 19:10:20 dist-test-slave-wd30 krb5kdc[14086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:20.884120 14174 init.cc:377] Logged in from keytab as kudu/127.0.203.61@KRBTEST.COM (short username kudu)
I20260620 19:10:20.884616 14174 webserver.cc:460] Webserver started at http://127.0.203.61:46437/ using document root <none> and password file <none>
I20260620 19:10:20.884837 14174 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:20.884904 14174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:20.885061 14174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:20.886123 14174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-1/data/instance:
uuid: "4e6a432acc1e497bba1aa445cb69d3e4"
format_stamp: "Formatted at 2026-06-20 19:10:20 on dist-test-slave-wd30"
server_key: "5f06ffc2b39dca9990c9f2bca0aa7bdd"
server_key_iv: "3da28411d83db56cce3f4ab3ce2f90d6"
server_key_version: "encryptionkey@0"
I20260620 19:10:20.886489 14174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-1/wal/instance:
uuid: "4e6a432acc1e497bba1aa445cb69d3e4"
format_stamp: "Formatted at 2026-06-20 19:10:20 on dist-test-slave-wd30"
server_key: "5f06ffc2b39dca9990c9f2bca0aa7bdd"
server_key_iv: "3da28411d83db56cce3f4ab3ce2f90d6"
server_key_version: "encryptionkey@0"
I20260620 19:10:20.887997 14174 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:10:20.888721 14189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:20.888896 14174 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:10:20.889003 14174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-1/wal
uuid: "4e6a432acc1e497bba1aa445cb69d3e4"
format_stamp: "Formatted at 2026-06-20 19:10:20 on dist-test-slave-wd30"
server_key: "5f06ffc2b39dca9990c9f2bca0aa7bdd"
server_key_iv: "3da28411d83db56cce3f4ab3ce2f90d6"
server_key_version: "encryptionkey@0"
I20260620 19:10:20.889086 14174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:20.895833 14167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } attempt: 2
I20260620 19:10:20.896238 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.896112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:34173 (local address 127.0.0.1:46852)
0620 19:10:20.896170 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:20.896207 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.61:34173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 19:10:20.896548 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.896451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:34173 (local address 127.0.0.1:46856)
0620 19:10:20.896497 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:20.896520 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.61:34173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 19:10:20.896629 14167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.61:34173: Network error: Client connection negotiation failed: client connection to 127.0.203.61:34173: connect: Connection refused (error 111)
I20260620 19:10:20.902352 14174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:20.904405 14174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:20.904590 14174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:20.908619 14174 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.61:34173
I20260620 19:10:20.908648 14241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.61:34173 every 8 connection(s)
I20260620 19:10:20.908994 14174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-1/data/info.pb
I20260620 19:10:20.909832 14242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:20.911406 14242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:10:20.915282 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 14174
I20260620 19:10:20.915373 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-1/wal/instance
I20260620 19:10:20.915570 812 external_mini_cluster.cc:1428] Setting key 752cd5e899b7e0b3bae3d8968a8051f7
Jun 20 19:10:20 dist-test-slave-wd30 krb5kdc[14086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.61@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:20.920290 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.912080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39168 (local address 127.0.203.62:36847)
0620 19:10:20.912154 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:20.912156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:20.912315 (+ 159us) server_negotiation.cc:407] Connection header received
0620 19:10:20.912739 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:20.912740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:20.912749 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:20.912800 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:20.913367 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.913833 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.914787 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.914992 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.915042 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:20.917376 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:20.917378 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:20.917378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:20.917386 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:20.918759 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:20.919318 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:20.919320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:20.919320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:20.919381 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:20.919652 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:20.919653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:20.919654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:20.919769 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:20.919844 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:20.920113 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 19:10:20.920201 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:10:20.920488 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.912030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36847 (local address 127.0.0.1:39168)
0620 19:10:20.912196 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:20.912231 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:20.912661 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:20.912850 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:20.912853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:20.913111 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:20.913301 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:20.913305 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.913951 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.913952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:20.914712 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:20.914714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.915064 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.915065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:20.915234 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:20.915777 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:20.915793 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:20.917275 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:20.918863 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:20.918870 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:20.918872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:20.919227 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:20.919469 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:20.919470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:20.919471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:20.919583 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:20.919836 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:20.919842 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:20.919983 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 19:10:20.920093 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 19:10:20.920243 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260620 19:10:20.921034 14242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.203.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.60@KRBTEST.COM" created.
Jun 20 19:10:20 dist-test-slave-wd30 krb5kdc[14086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.61@KRBTEST.COM for kudu/127.0.203.61@KRBTEST.COM
I20260620 19:10:20.929358 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.921382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46872 (local address 127.0.203.61:34173)
0620 19:10:20.921538 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:20.921542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:20.921563 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:10:20.921579 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:20.921581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:20.921591 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:20.921670 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:20.921930 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.922609 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.923393 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.923681 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.923753 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:20.926487 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:20.926496 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:20.926497 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:20.926504 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:20.928157 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:20.928545 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:20.928548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:20.928548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:20.928598 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:20.928796 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:20.928796 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:20.928796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:20.928923 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:20.928982 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:20.929152 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 19:10:20.929275 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":130,"thread_start_us":104,"threads_started":1}
I20260620 19:10:20.929420 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.921296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:34173 (local address 127.0.0.1:46872)
0620 19:10:20.921344 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:20.921355 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:20.921393 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:20.921671 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:20.921672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:20.921811 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:20.921851 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:20.921853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.922708 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.922709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:20.923307 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:20.923309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.923761 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.923762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:20.923826 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:20.924343 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:20.924347 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:20.926400 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:20.928241 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:20.928242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:20.928243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:20.928466 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:20.928665 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:20.928666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:20.928666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:20.928737 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:20.928989 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:20.928990 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:20.929039 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 19:10:20.929112 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 19:10:20.929228 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 19:10:20.930231 14242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:10:20.930605 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.930467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:40385 (local address 127.0.0.1:56778)
0620 19:10:20.930518 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:20.930574 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:40385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 19:10:20.930757 14192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.203.60:40385: connect: Connection refused (error 111)
I20260620 19:10:20.931231 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.931148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:40385 (local address 127.0.0.1:56786)
0620 19:10:20.931179 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:20.931206 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:40385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260620 19:10:20.931329 14242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.60:40385: Network error: Client connection negotiation failed: client connection to 127.0.203.60:40385: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.60.keytab.
Entry for principal kudu/127.0.203.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.60.keytab.
I20260620 19:10:20.941370 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.60:40385
--webserver_interface=127.0.203.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.60.keytab
--principal=kudu/127.0.203.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:36847,127.0.203.61:34173,127.0.203.60:40385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:10:20.980582 14242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } attempt: 1
I20260620 19:10:20.981189 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.981052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:40385 (local address 127.0.0.1:56798)
0620 19:10:20.981119 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:20.981160 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:40385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 19:10:20.981519 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.981427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:40385 (local address 127.0.0.1:56810)
0620 19:10:20.981471 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:20.981495 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:40385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 19:10:20.981613 14242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.60:40385: Network error: Client connection negotiation failed: client connection to 127.0.203.60:40385: connect: Connection refused (error 111)
I20260620 19:10:20.987756 14167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } attempt: 3
Jun 20 19:10:20 dist-test-slave-wd30 krb5kdc[14086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for kudu/127.0.203.61@KRBTEST.COM
I20260620 19:10:20.994863 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.988132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46884 (local address 127.0.203.61:34173)
0620 19:10:20.988194 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:20.988196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:20.988207 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:10:20.988269 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:20.988270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:20.988281 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:20.988353 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:20.988694 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.989210 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.989929 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.990150 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.990226 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:20.992419 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:20.992420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:20.992421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:20.992435 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:20.993823 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:20.994203 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:20.994204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:20.994204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:20.994253 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:20.994439 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:20.994439 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:20.994440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:20.994537 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:20.994585 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:20.994747 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 19:10:20.994805 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:10:20.994920 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.988057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:34173 (local address 127.0.0.1:46884)
0620 19:10:20.988111 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:20.988123 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:20.988199 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:20.988357 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:20.988358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:20.988535 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:20.988604 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:20.988606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.989294 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.989295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:20.989829 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:20.989830 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.990223 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.990224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:20.990305 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:20.990803 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:20.990807 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:20.992279 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:20.993907 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:20.993908 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:20.993908 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:20.994123 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:20.994312 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:20.994313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:20.994314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:20.994379 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:20.994598 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:20.994599 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:20.994650 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 19:10:20.994722 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 19:10:20.994824 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 19:10:20.995347 14167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:10:20.995726 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.995598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:40385 (local address 127.0.0.1:56824)
0620 19:10:20.995659 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:20.995694 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:40385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 19:10:20.996096 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.996004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:40385 (local address 127.0.0.1:56828)
0620 19:10:20.996051 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:20.996078 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:40385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 19:10:20.996192 14167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.60:40385: Network error: Client connection negotiation failed: client connection to 127.0.203.60:40385: connect: Connection refused (error 111)
W20260620 19:10:21.023937 14249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:21.024140 14249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:21.024163 14249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:21.025302 14249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:21.025341 14249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:21.025355 14249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:21.025368 14249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:21.025380 14249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:21.026849 14249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:36847,127.0.203.61:34173,127.0.203.60:40385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.60.keytab
--principal=kudu/127.0.203.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.60:40385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.203.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:21.027096 14249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:21.027304 14167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } attempt: 1
I20260620 19:10:21.027417 14249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:10:21.027747 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:21.027606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:40385 (local address 127.0.0.1:56838)
0620 19:10:21.027671 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:21.027714 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:40385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 19:10:21.028136 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:21.028053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:40385 (local address 127.0.0.1:56844)
0620 19:10:21.028095 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:21.028119 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:40385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 19:10:21.028258 14167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.60:40385: Network error: Client connection negotiation failed: client connection to 127.0.203.60:40385: connect: Connection refused (error 111)
W20260620 19:10:21.032481 14257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:21.032496 14255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:21.032526 14249 server_base.cc:1034] running on GCE node
W20260620 19:10:21.032505 14254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:21.032972 14249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:21.033303 14249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:21.034493 14249 hybrid_clock.cc:648] HybridClock initialized: now 1781982621034480 us; error 47 us; skew 500 ppm
Jun 20 19:10:21 dist-test-slave-wd30 krb5kdc[14086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:21.037278 14249 init.cc:377] Logged in from keytab as kudu/127.0.203.60@KRBTEST.COM (short username kudu)
I20260620 19:10:21.037840 14249 webserver.cc:460] Webserver started at http://127.0.203.60:39717/ using document root <none> and password file <none>
I20260620 19:10:21.038079 14249 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:21.038151 14249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:21.038317 14249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:21.039618 14249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-2/data/instance:
uuid: "47246a1049d04684a8a7a4d6398e0fee"
format_stamp: "Formatted at 2026-06-20 19:10:21 on dist-test-slave-wd30"
server_key: "b325c28ce537927513b8555744ab8185"
server_key_iv: "c5303adf0c32a96e28c75fe92f2004d0"
server_key_version: "encryptionkey@0"
I20260620 19:10:21.039999 14249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-2/wal/instance:
uuid: "47246a1049d04684a8a7a4d6398e0fee"
format_stamp: "Formatted at 2026-06-20 19:10:21 on dist-test-slave-wd30"
server_key: "b325c28ce537927513b8555744ab8185"
server_key_iv: "c5303adf0c32a96e28c75fe92f2004d0"
server_key_version: "encryptionkey@0"
I20260620 19:10:21.041545 14249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:10:21.042387 14264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:21.042590 14249 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:21.042706 14249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-2/wal
uuid: "47246a1049d04684a8a7a4d6398e0fee"
format_stamp: "Formatted at 2026-06-20 19:10:21 on dist-test-slave-wd30"
server_key: "b325c28ce537927513b8555744ab8185"
server_key_iv: "c5303adf0c32a96e28c75fe92f2004d0"
server_key_version: "encryptionkey@0"
I20260620 19:10:21.042788 14249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:21.049752 14242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } attempt: 2
I20260620 19:10:21.050366 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:21.050216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:40385 (local address 127.0.0.1:56848)
0620 19:10:21.050292 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:21.050334 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:40385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 19:10:21.050776 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:21.050688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:40385 (local address 127.0.0.1:56856)
0620 19:10:21.050732 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:21.050757 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:40385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 19:10:21.050923 14242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.60:40385: Network error: Client connection negotiation failed: client connection to 127.0.203.60:40385: connect: Connection refused (error 111)
I20260620 19:10:21.058329 14249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:21.060614 14249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:21.060768 14249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:21.064831 14249 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.60:40385
I20260620 19:10:21.064863 14316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.60:40385 every 8 connection(s)
I20260620 19:10:21.065284 14249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-2/data/info.pb
I20260620 19:10:21.066072 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 14249
I20260620 19:10:21.066205 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982468536772-812-0/minicluster-data/master-2/wal/instance
I20260620 19:10:21.066246 14317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:21.066414 812 external_mini_cluster.cc:1428] Setting key 990fe8a6cf1db85f39927f7d6e81abaf
I20260620 19:10:21.068681 14317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:10:21 dist-test-slave-wd30 krb5kdc[14086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.60@KRBTEST.COM
Jun 20 19:10:21 dist-test-slave-wd30 krb5kdc[14086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.60@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:21.077491 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:21.066945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56858 (local address 127.0.203.60:40385)
0620 19:10:21.067179 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:21.067194 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:21.067221 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:10:21.067770 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:21.067777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:21.067826 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:21.067941 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:21.068710 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.069905 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.070862 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.071503 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.071613 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:21.073985 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:21.073993 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:21.073993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:21.074000 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:21.075755 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.076175 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.076179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.076180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.076280 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.076496 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.076497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.076497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.076811 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:21.076906 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:21.077101 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 19:10:21.077246 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
I20260620 19:10:21.078019 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:21.069369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39176 (local address 127.0.203.62:36847)
0620 19:10:21.069437 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:21.069439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:21.069607 (+ 168us) server_negotiation.cc:407] Connection header received
0620 19:10:21.069667 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:21.069668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:21.069677 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:21.069725 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:21.070248 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.070927 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.071928 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.072127 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.072199 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:21.074498 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:21.074499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:21.074499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:21.074506 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:21.076663 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.077088 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.077089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.077089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.077148 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.077451 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.077452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.077452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.077596 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:21.077666 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:21.077848 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 19:10:21.077930 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:10:21.078033 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:21.069306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36847 (local address 127.0.0.1:39176)
0620 19:10:21.069498 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:21.069538 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:21.069608 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:21.069759 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:21.069761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:21.070104 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:21.070170 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:21.070171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.071008 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.071009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:21.071836 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:21.071838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.072219 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.072220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:21.072297 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:21.072762 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:21.072770 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:21.074399 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:21.076742 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:21.076747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:21.076749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:21.077013 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:21.077320 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:21.077321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:21.077321 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:21.077388 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:21.077659 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:21.077663 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:21.077713 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 19:10:21.077819 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 19:10:21.077930 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":111,"threads_started":1}
I20260620 19:10:21.078711 14317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:10:21 dist-test-slave-wd30 krb5kdc[14086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.60@KRBTEST.COM for kudu/127.0.203.61@KRBTEST.COM
Jun 20 19:10:21 dist-test-slave-wd30 krb5kdc[14086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:21.085912 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:21.078963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:34173 (local address 127.0.0.1:46888)
0620 19:10:21.079009 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:21.079019 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:21.079052 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:21.079210 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:21.079211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:21.079344 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:21.079372 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:21.079373 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.079980 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.079981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:21.080460 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:21.080461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.080911 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.080912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:21.080971 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:21.081405 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:21.081409 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:21.082617 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:21.084890 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:21.084891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:21.084891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:21.085056 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:21.085285 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:21.085286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:21.085287 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:21.085341 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:21.085591 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:21.085592 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:21.085633 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 19:10:21.085722 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 19:10:21.085820 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 19:10:21.085912 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:21.079042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46888 (local address 127.0.203.61:34173)
0620 19:10:21.079098 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:21.079100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:21.079108 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:10:21.079151 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:21.079152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:21.079161 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:21.079209 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:21.079449 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.079905 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.080542 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.080799 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.080860 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:21.082698 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:21.082699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:21.082700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:21.082707 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:21.084791 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.085140 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.085141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.085142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.085195 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.085412 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.085413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.085413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.085525 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:21.085599 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:21.085746 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 19:10:21.085820 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 19:10:21.086102 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:21.078498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39192 (local address 127.0.203.62:36847)
0620 19:10:21.078537 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:21.078539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:21.078548 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:10:21.078605 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:21.078606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:21.078615 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:21.078693 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:21.079291 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.079792 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.080483 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.080669 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.080742 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:21.083338 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:21.083340 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:21.083340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:21.083355 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:21.084973 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.085297 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.085298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.085299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.085362 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.085560 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.085561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.085561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.085668 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:21.085870 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:21.085906 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 19:10:21.085987 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:10:21.086865 14317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:10:21 dist-test-slave-wd30 krb5kdc[14086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.61@KRBTEST.COM
Jun 20 19:10:21 dist-test-slave-wd30 krb5kdc[14086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.60@KRBTEST.COM for kudu/127.0.203.60@KRBTEST.COM
I20260620 19:10:21.094105 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:21.086785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46898 (local address 127.0.203.61:34173)
0620 19:10:21.086836 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:21.086837 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:21.086893 (+ 56us) server_negotiation.cc:407] Connection header received
0620 19:10:21.086961 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:21.086962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:21.086972 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:21.087029 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:21.087707 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.088124 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.088774 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.088981 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.089055 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:21.091195 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:21.091197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:21.091197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:21.091206 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:21.092947 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.093290 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.093291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.093291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.093350 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.093555 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.093556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.093556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.093674 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:21.093878 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:21.093917 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 19:10:21.093989 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 19:10:21.094239 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:21.087230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.0.203.60:40385)
0620 19:10:21.087266 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:21.087268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:21.087276 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:10:21.087320 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:21.087321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:21.087329 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:21.087397 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:21.087647 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.088141 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.089103 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.089412 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.089494 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:21.091641 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:21.091642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:21.091643 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:21.091650 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:21.093152 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.093462 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.093463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.093463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.093513 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.093712 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.093713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.093713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.093821 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:21.093895 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:21.094018 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 19:10:21.094143 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":20}
I20260620 19:10:21.094332 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:21.087182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:40385 (local address 127.0.0.1:56860)
0620 19:10:21.087223 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:21.087233 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:21.087268 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:21.087400 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:21.087401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:21.087551 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:21.087578 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:21.087579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.088230 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.088230 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:21.088688 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:21.088689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.089489 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.089489 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:21.089530 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:21.089925 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:21.089930 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:21.091560 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:21.093224 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:21.093226 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:21.093226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:21.093390 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:21.093586 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:21.093587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:21.093587 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:21.093640 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:21.093888 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:21.093889 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:21.093926 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 19:10:21.093995 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 19:10:21.094093 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 19:10:21.095798 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:10:21.096167 14317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee: Bootstrap starting.
I20260620 19:10:21.097074 14317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:21.097450 14317 log.cc:824] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:21.098256 14317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee: No bootstrap required, opened a new log
I20260620 19:10:21.100508 14317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2114d3ed844999fc36f2265101eff" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } } peers { permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } } peers { permanent_uuid: "47246a1049d04684a8a7a4d6398e0fee" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } }
I20260620 19:10:21.100786 14317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:21.100876 14317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47246a1049d04684a8a7a4d6398e0fee, State: Initialized, Role: FOLLOWER
I20260620 19:10:21.101080 14317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2114d3ed844999fc36f2265101eff" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } } peers { permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } } peers { permanent_uuid: "47246a1049d04684a8a7a4d6398e0fee" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } }
I20260620 19:10:21.101445 14317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:21.102654 14330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2114d3ed844999fc36f2265101eff" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } } peers { permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } } peers { permanent_uuid: "47246a1049d04684a8a7a4d6398e0fee" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } } }
I20260620 19:10:21.103256 14330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:10:21.103379 14167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } attempt: 2
I20260620 19:10:21.105783 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:21.097480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46914 (local address 127.0.203.61:34173)
0620 19:10:21.097517 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:21.097519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:21.098219 (+ 700us) server_negotiation.cc:407] Connection header received
0620 19:10:21.098242 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:21.098243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:21.098254 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:21.098315 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:21.099086 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.099759 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.100712 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.100995 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.101066 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:21.102625 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:21.102626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:21.102626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:21.102634 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:21.104274 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.104767 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.104768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.104769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.104825 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.105030 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.105031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.105031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.105162 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:21.105249 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:21.105618 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 19:10:21.105699 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:10:21.105746 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:21.097323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56862 (local address 127.0.203.60:40385)
0620 19:10:21.097357 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:21.097359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:21.097536 (+ 177us) server_negotiation.cc:407] Connection header received
0620 19:10:21.097559 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:21.097559 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:21.097571 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:21.097623 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:21.098460 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.099214 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.100308 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.100596 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.100654 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:21.102350 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:21.102352 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:21.102352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:21.102359 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:21.104091 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.104864 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.104865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.104866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.104923 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.105144 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.105145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.105145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.105289 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:21.105378 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:21.105575 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 19:10:21.105661 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:10:21.105746 14249 master_runner.cc:186] Error getting master registration for 127.0.203.62:36847: OK, instance_id { permanent_uuid: "54f2114d3ed844999fc36f2265101eff" instance_seqno: 1781982620760653 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:10:21.108263 14249 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 19:10:21.103524 14341 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:10:21.109411 14341 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:10:21.110949 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:21.097233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39204 (local address 127.0.203.62:36847)
0620 19:10:21.097300 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:21.097302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:21.097606 (+ 304us) server_negotiation.cc:407] Connection header received
0620 19:10:21.097678 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:21.097679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:21.097690 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:21.097734 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:21.098461 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.099244 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.100313 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.100619 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.100655 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:21.104067 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:21.104068 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:21.104069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:21.104076 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:21.109321 (+ 5245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.109856 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.109857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.109858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.109953 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.110178 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.110179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.110179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.110324 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:21.110437 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:21.110662 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 19:10:21.110748 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 19:10:21 dist-test-slave-wd30 krb5kdc[14086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for kudu/127.0.203.60@KRBTEST.COM
I20260620 19:10:21.114069 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:21.103851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:40385 (local address 127.0.0.1:56872)
0620 19:10:21.103902 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:21.103912 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:21.103948 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:21.106470 (+ 2522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:21.106471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:21.106632 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:21.106675 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:21.106677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.107682 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.107683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:21.108433 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:21.108436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.109340 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.109341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:21.109407 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:21.109818 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:21.109824 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:21.111509 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:21.113181 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:21.113182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:21.113183 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:21.113354 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:21.113556 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:21.113556 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:21.113556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:21.113603 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:21.113856 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:21.113857 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:21.113890 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 19:10:21.113939 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:10:21.114007 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 19:10:21.114127 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:21.104205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.0.203.60:40385)
0620 19:10:21.105856 (+ 1651us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:21.105858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:21.105865 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:10:21.105882 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:21.105882 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:21.105891 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:21.105943 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:21.106877 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.107583 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.108918 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.109220 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.109288 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:21.111595 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:21.111596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:21.111597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:21.111604 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:21.113103 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.113426 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.113427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.113428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.113488 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.113662 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.113662 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.113663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.113790 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:21.113867 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:21.113965 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 19:10:21.114038 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1638,"thread_start_us":81,"threads_started":1}
I20260620 19:10:21.116020 14167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff: Bootstrap starting.
I20260620 19:10:21.116755 14167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:21.117048 14167 log.cc:824] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:21.117906 14167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff: No bootstrap required, opened a new log
I20260620 19:10:21.119938 14167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2114d3ed844999fc36f2265101eff" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } } peers { permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } } peers { permanent_uuid: "47246a1049d04684a8a7a4d6398e0fee" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } }
I20260620 19:10:21.120082 14167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:21.120121 14167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f2114d3ed844999fc36f2265101eff, State: Initialized, Role: FOLLOWER
I20260620 19:10:21.120240 14167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2114d3ed844999fc36f2265101eff" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } } peers { permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } } peers { permanent_uuid: "47246a1049d04684a8a7a4d6398e0fee" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } }
I20260620 19:10:21.120504 14345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2114d3ed844999fc36f2265101eff" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } } peers { permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } } peers { permanent_uuid: "47246a1049d04684a8a7a4d6398e0fee" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } } }
I20260620 19:10:21.120553 14167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:21.120615 14345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [sys.catalog]: This master's current role is: FOLLOWER
W20260620 19:10:21.122677 14356 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:10:21.122779 14356 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:10:21.123703 14099 master_runner.cc:186] Error getting master registration for 127.0.203.61:34173: OK, instance_id { permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" instance_seqno: 1781982620907466 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:10:21.125952 14099 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:10:21.142107 14242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } attempt: 3
Jun 20 19:10:21 dist-test-slave-wd30 krb5kdc[14086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.61@KRBTEST.COM for kudu/127.0.203.60@KRBTEST.COM
I20260620 19:10:21.150628 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:21.142410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:40385 (local address 127.0.0.1:56886)
0620 19:10:21.142465 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:21.142478 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:21.142555 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:21.142726 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:21.142727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:21.142923 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:21.142984 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:21.142986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.143952 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.143953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:21.144775 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:21.144777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.145272 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.145273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:21.145336 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:21.145895 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:21.145901 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:21.147575 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:21.149480 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:21.149481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:21.149482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:21.149700 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:21.149926 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:21.149927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:21.149927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:21.149997 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:21.150285 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:21.150286 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:21.150340 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 19:10:21.150419 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 19:10:21.150520 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 19:10:21.150631 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:21.142513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.0.203.60:40385)
0620 19:10:21.142575 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:21.142577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:21.142586 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:10:21.142633 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:21.142634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:21.142645 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:21.143054 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:21.143088 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.143863 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.144875 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:21.145191 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:21.145274 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:21.147673 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:21.147675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:21.147675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:21.147683 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:21.149382 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.149780 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.149781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.149781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.149852 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:21.150063 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:21.150064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:21.150065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:21.150205 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:21.150275 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:21.150456 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 19:10:21.150538 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:10:21.152822 14242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4: Bootstrap starting.
I20260620 19:10:21.153602 14242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:21.153918 14242 log.cc:824] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:21.154685 14242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4: No bootstrap required, opened a new log
I20260620 19:10:21.156800 14242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2114d3ed844999fc36f2265101eff" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } } peers { permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } } peers { permanent_uuid: "47246a1049d04684a8a7a4d6398e0fee" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } }
I20260620 19:10:21.156950 14242 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:21.156985 14242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6a432acc1e497bba1aa445cb69d3e4, State: Initialized, Role: FOLLOWER
I20260620 19:10:21.157102 14242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2114d3ed844999fc36f2265101eff" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } } peers { permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } } peers { permanent_uuid: "47246a1049d04684a8a7a4d6398e0fee" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } }
I20260620 19:10:21.157411 14242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:21.157397 14359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2114d3ed844999fc36f2265101eff" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } } peers { permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } } peers { permanent_uuid: "47246a1049d04684a8a7a4d6398e0fee" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } } }
I20260620 19:10:21.157595 14359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 19:10:21.159348 14370 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:10:21.159451 14370 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:10:21.161473 14174 master_runner.cc:263] No leader master found from master 4e6a432acc1e497bba1aa445cb69d3e4
I20260620 19:10:21.161520 14174 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 19:10:21.258539 14120 tablet.cc:2369] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:10:21.269369 14345 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:21.269533 14345 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2114d3ed844999fc36f2265101eff" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } } peers { permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } } peers { permanent_uuid: "47246a1049d04684a8a7a4d6398e0fee" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } }
I20260620 19:10:21.269816 14345 leader_election.cc:288] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e6a432acc1e497bba1aa445cb69d3e4 (127.0.203.61:34173), 47246a1049d04684a8a7a4d6398e0fee (127.0.203.60:40385)
I20260620 19:10:21.270125 14291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54f2114d3ed844999fc36f2265101eff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47246a1049d04684a8a7a4d6398e0fee" is_pre_election: true
I20260620 19:10:21.270140 14216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54f2114d3ed844999fc36f2265101eff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" is_pre_election: true
I20260620 19:10:21.270292 14216 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54f2114d3ed844999fc36f2265101eff in term 0.
I20260620 19:10:21.270296 14291 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54f2114d3ed844999fc36f2265101eff in term 0.
I20260620 19:10:21.270473 14115 leader_election.cc:302] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e6a432acc1e497bba1aa445cb69d3e4, 54f2114d3ed844999fc36f2265101eff; no voters:
I20260620 19:10:21.270646 14345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:21.270711 14345 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:21.270743 14345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:21.271622 14345 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2114d3ed844999fc36f2265101eff" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } } peers { permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } } peers { permanent_uuid: "47246a1049d04684a8a7a4d6398e0fee" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } }
I20260620 19:10:21.271768 14345 leader_election.cc:288] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [CANDIDATE]: Term 1 election: Requested vote from peers 4e6a432acc1e497bba1aa445cb69d3e4 (127.0.203.61:34173), 47246a1049d04684a8a7a4d6398e0fee (127.0.203.60:40385)
I20260620 19:10:21.271976 14291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54f2114d3ed844999fc36f2265101eff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47246a1049d04684a8a7a4d6398e0fee"
I20260620 19:10:21.271991 14216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54f2114d3ed844999fc36f2265101eff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6a432acc1e497bba1aa445cb69d3e4"
I20260620 19:10:21.272064 14216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:21.272065 14291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:21.272918 14291 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54f2114d3ed844999fc36f2265101eff in term 1.
I20260620 19:10:21.272918 14216 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54f2114d3ed844999fc36f2265101eff in term 1.
I20260620 19:10:21.273089 14115 leader_election.cc:302] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e6a432acc1e497bba1aa445cb69d3e4, 54f2114d3ed844999fc36f2265101eff; no voters:
I20260620 19:10:21.273197 14345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:21.273327 14345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [term 1 LEADER]: Becoming Leader. State: Replica: 54f2114d3ed844999fc36f2265101eff, State: Running, Role: LEADER
I20260620 19:10:21.273409 14345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2114d3ed844999fc36f2265101eff" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } } peers { permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } } peers { permanent_uuid: "47246a1049d04684a8a7a4d6398e0fee" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } }
I20260620 19:10:21.273856 14372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54f2114d3ed844999fc36f2265101eff. Latest consensus state: current_term: 1 leader_uuid: "54f2114d3ed844999fc36f2265101eff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2114d3ed844999fc36f2265101eff" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } } peers { permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } } peers { permanent_uuid: "47246a1049d04684a8a7a4d6398e0fee" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } } }
I20260620 19:10:21.273941 14372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:21.274142 14374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:21.274331 14374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:21.275735 14291 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee [term 1 FOLLOWER]: Refusing update from remote peer 54f2114d3ed844999fc36f2265101eff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:21.275725 14216 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4 [term 1 FOLLOWER]: Refusing update from remote peer 54f2114d3ed844999fc36f2265101eff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:21.276049 14372 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [LEADER]: Connected to new peer: Peer: permanent_uuid: "47246a1049d04684a8a7a4d6398e0fee" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:21.276129 14345 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:21.277441 14359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54f2114d3ed844999fc36f2265101eff. Latest consensus state: current_term: 1 leader_uuid: "54f2114d3ed844999fc36f2265101eff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2114d3ed844999fc36f2265101eff" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } } peers { permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } } peers { permanent_uuid: "47246a1049d04684a8a7a4d6398e0fee" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } } }
I20260620 19:10:21.277530 14359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:10:21.277477 14330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54f2114d3ed844999fc36f2265101eff. Latest consensus state: current_term: 1 leader_uuid: "54f2114d3ed844999fc36f2265101eff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2114d3ed844999fc36f2265101eff" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } } peers { permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } } peers { permanent_uuid: "47246a1049d04684a8a7a4d6398e0fee" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } } }
I20260620 19:10:21.277590 14330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:10:21.278623 14345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54f2114d3ed844999fc36f2265101eff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2114d3ed844999fc36f2265101eff" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } } peers { permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } } peers { permanent_uuid: "47246a1049d04684a8a7a4d6398e0fee" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } } }
I20260620 19:10:21.278724 14345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:21.278748 14330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54f2114d3ed844999fc36f2265101eff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2114d3ed844999fc36f2265101eff" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } } peers { permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } } peers { permanent_uuid: "47246a1049d04684a8a7a4d6398e0fee" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } } }
I20260620 19:10:21.278843 14330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:10:21.278831 14375 mvcc.cc:205] Tried to move back new op lower bound from 7299000816743469056 to 7299000816736501760. Current Snapshot: MvccSnapshot[applied={T|T < 7299000816743469056}]
I20260620 19:10:21.279063 14345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54f2114d3ed844999fc36f2265101eff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2114d3ed844999fc36f2265101eff" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } } peers { permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } } peers { permanent_uuid: "47246a1049d04684a8a7a4d6398e0fee" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } } }
I20260620 19:10:21.279088 14376 mvcc.cc:205] Tried to move back new op lower bound from 7299000816743469056 to 7299000816736501760. Current Snapshot: MvccSnapshot[applied={T|T < 7299000816743469056}]
I20260620 19:10:21.279162 14345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:21.279654 14374 catalog_manager.cc:1347] Generated new cluster ID: f4df6c7769f84fe5887ddbf70be13c6f
I20260620 19:10:21.279681 14377 mvcc.cc:205] Tried to move back new op lower bound from 7299000816743469056 to 7299000816736501760. Current Snapshot: MvccSnapshot[applied={T|T < 7299000816743469056}]
I20260620 19:10:21.279717 14374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:21.279779 14359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54f2114d3ed844999fc36f2265101eff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2114d3ed844999fc36f2265101eff" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36847 } } peers { permanent_uuid: "4e6a432acc1e497bba1aa445cb69d3e4" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34173 } } peers { permanent_uuid: "47246a1049d04684a8a7a4d6398e0fee" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 40385 } } }
I20260620 19:10:21.279841 14359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:10:21.293843 14374 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:21.294596 14374 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:21.305591 14374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54f2114d3ed844999fc36f2265101eff: Generated new TSK 0
I20260620 19:10:21.305809 14374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:10:22.109931 14341 catalog_manager.cc:1259] Loaded cluster ID: f4df6c7769f84fe5887ddbf70be13c6f
I20260620 19:10:22.110046 14341 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee: loading cluster ID for follower catalog manager: success
I20260620 19:10:22.110877 14341 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee: acquiring CA information for follower catalog manager: success
I20260620 19:10:22.111229 14341 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 47246a1049d04684a8a7a4d6398e0fee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:10:22.159909 14370 catalog_manager.cc:1259] Loaded cluster ID: f4df6c7769f84fe5887ddbf70be13c6f
I20260620 19:10:22.160002 14370 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4: loading cluster ID for follower catalog manager: success
I20260620 19:10:22.160859 14370 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4: acquiring CA information for follower catalog manager: success
I20260620 19:10:22.161188 14370 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4e6a432acc1e497bba1aa445cb69d3e4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:10:22.368678 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 14099
I20260620 19:10:22.373956 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 14174
I20260620 19:10:22.378755 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 14249
2026-06-20T19:10:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2916 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:10:22 dist-test-slave-wd30 krb5kdc[14389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:10:22 dist-test-slave-wd30 krb5kdc[14389](info): set up 2 sockets
Jun 20 19:10:22 dist-test-slave-wd30 krb5kdc[14389](info): commencing operation
krb5kdc: starting...
W20260620 19:10:23.443272 812 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:10:23 dist-test-slave-wd30 krb5kdc[14389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:10:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:10:23.562398 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:35475
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:34587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:35475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:23.643573 14402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:23.643893 14402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:23.643936 14402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:23.645162 14402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:23.645208 14402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:23.645222 14402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:23.645236 14402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:23.645247 14402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:23.646677 14402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:34587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:35475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:35475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:23.646930 14402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:23.647249 14402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:23.652081 14407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:23.652117 14408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:23.652092 14410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:23.652580 14402 server_base.cc:1034] running on GCE node
I20260620 19:10:23.652835 14402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:23.653127 14402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:23.654280 14402 hybrid_clock.cc:648] HybridClock initialized: now 1781982623654261 us; error 40 us; skew 500 ppm
Jun 20 19:10:23 dist-test-slave-wd30 krb5kdc[14389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:23.656852 14402 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:10:23.657320 14402 webserver.cc:460] Webserver started at http://127.0.203.62:39557/ using document root <none> and password file <none>
I20260620 19:10:23.657564 14402 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:23.657637 14402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:23.657783 14402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:23.658862 14402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "d4e0fffaf7014779a7d7538f470cdfa9"
format_stamp: "Formatted at 2026-06-20 19:10:23 on dist-test-slave-wd30"
server_key: "55e50b88c9cc388c76db416c5b38e859"
server_key_iv: "7060b30adeda1eaaf404a821a5843077"
server_key_version: "encryptionkey@0"
I20260620 19:10:23.659279 14402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "d4e0fffaf7014779a7d7538f470cdfa9"
format_stamp: "Formatted at 2026-06-20 19:10:23 on dist-test-slave-wd30"
server_key: "55e50b88c9cc388c76db416c5b38e859"
server_key_iv: "7060b30adeda1eaaf404a821a5843077"
server_key_version: "encryptionkey@0"
I20260620 19:10:23.660583 14402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:23.661298 14417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:23.661476 14402 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:10:23.661576 14402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "d4e0fffaf7014779a7d7538f470cdfa9"
format_stamp: "Formatted at 2026-06-20 19:10:23 on dist-test-slave-wd30"
server_key: "55e50b88c9cc388c76db416c5b38e859"
server_key_iv: "7060b30adeda1eaaf404a821a5843077"
server_key_version: "encryptionkey@0"
I20260620 19:10:23.661659 14402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:23.671366 14402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:23.673722 14402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:23.673898 14402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:23.678053 14402 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:35475
I20260620 19:10:23.678092 14469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:35475 every 8 connection(s)
I20260620 19:10:23.678447 14402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:23.679397 14470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:23.681723 14470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9: Bootstrap starting.
I20260620 19:10:23.682265 14470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:23.682494 14470 log.cc:824] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:23.683265 14470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9: No bootstrap required, opened a new log
I20260620 19:10:23.685065 14470 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e0fffaf7014779a7d7538f470cdfa9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35475 } }
I20260620 19:10:23.685217 14470 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:23.685242 14470 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e0fffaf7014779a7d7538f470cdfa9, State: Initialized, Role: FOLLOWER
I20260620 19:10:23.685406 14470 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e0fffaf7014779a7d7538f470cdfa9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35475 } }
I20260620 19:10:23.685500 14470 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:23.685525 14470 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:23.685557 14470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:23.686091 14470 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e0fffaf7014779a7d7538f470cdfa9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35475 } }
I20260620 19:10:23.686182 14470 leader_election.cc:302] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4e0fffaf7014779a7d7538f470cdfa9; no voters:
I20260620 19:10:23.686316 14470 leader_election.cc:288] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:23.686380 14473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:23.686560 14473 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9 [term 1 LEADER]: Becoming Leader. State: Replica: d4e0fffaf7014779a7d7538f470cdfa9, State: Running, Role: LEADER
I20260620 19:10:23.686643 14470 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:23.686702 14473 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e0fffaf7014779a7d7538f470cdfa9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35475 } }
I20260620 19:10:23.687071 14475 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4e0fffaf7014779a7d7538f470cdfa9. Latest consensus state: current_term: 1 leader_uuid: "d4e0fffaf7014779a7d7538f470cdfa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e0fffaf7014779a7d7538f470cdfa9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35475 } } }
I20260620 19:10:23.687183 14475 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:23.687230 14474 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4e0fffaf7014779a7d7538f470cdfa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e0fffaf7014779a7d7538f470cdfa9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35475 } } }
I20260620 19:10:23.687288 14474 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:23.687479 14480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:23.687570 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 14402
I20260620 19:10:23.687668 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:10:23.687885 812 external_mini_cluster.cc:1428] Setting key 7fcf21a2e3e612a65cf16b467112c273
I20260620 19:10:23.688086 14480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:23.689834 14480 catalog_manager.cc:1347] Generated new cluster ID: faf0a342d9cb4f8492904a203fa99ad9
I20260620 19:10:23.689888 14480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:10:23 dist-test-slave-wd30 krb5kdc[14389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:23.700390 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:23.688550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.0.203.62:35475)
0620 19:10:23.688921 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:23.688931 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:23.688968 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:10:23.689092 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:23.689099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:23.689158 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:23.689301 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:23.690176 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:23.691593 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:23.692690 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:23.693298 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:23.693387 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:23.696429 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:23.696433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:23.696434 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:23.696447 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:23.698498 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:23.698928 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:23.698932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:23.698932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:23.699060 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:23.699295 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:23.699296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:23.699297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:23.699607 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:23.699763 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:23.699900 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 19:10:23.700153 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":136,"threads_started":1}
I20260620 19:10:23.701337 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:10:23.712127 14480 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:23.712947 14480 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:23.723080 14480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4e0fffaf7014779a7d7538f470cdfa9: Generated new TSK 0
I20260620 19:10:23.723354 14480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:10:23.755357 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 14402
2026-06-20T19:10:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1375 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:10:23 dist-test-slave-wd30 krb5kdc[14497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:10:23 dist-test-slave-wd30 krb5kdc[14497](info): set up 2 sockets
Jun 20 19:10:23 dist-test-slave-wd30 krb5kdc[14497](info): commencing operation
krb5kdc: starting...
W20260620 19:10:24.799306 812 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:10:24 dist-test-slave-wd30 krb5kdc[14497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:10:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:10:24.917985 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:42193
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:42193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:24.999053 14510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:24.999295 14510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:24.999320 14510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:25.000509 14510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:25.000553 14510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:25.000568 14510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:25.000581 14510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:25.000597 14510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:25.002101 14510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:42193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:42193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:25.002350 14510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:25.002581 14510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:25.007432 14515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:25.007452 14518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:25.007432 14516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:25.007732 14510 server_base.cc:1034] running on GCE node
I20260620 19:10:25.008003 14510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:25.008287 14510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:25.009472 14510 hybrid_clock.cc:648] HybridClock initialized: now 1781982625009468 us; error 37 us; skew 500 ppm
Jun 20 19:10:25 dist-test-slave-wd30 krb5kdc[14497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:25.012025 14510 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:10:25.012512 14510 webserver.cc:460] Webserver started at http://127.0.203.62:38331/ using document root <none> and password file <none>
I20260620 19:10:25.012761 14510 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:25.012835 14510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:25.012986 14510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:25.014045 14510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "05e41ecd2c7045cf8ee6e171c568489f"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-wd30"
server_key: "06e522be9fda5f7f0b4e107360c0c8d9"
server_key_iv: "896517d6d15b7b75522844025f42dc2b"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.014403 14510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "05e41ecd2c7045cf8ee6e171c568489f"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-wd30"
server_key: "06e522be9fda5f7f0b4e107360c0c8d9"
server_key_iv: "896517d6d15b7b75522844025f42dc2b"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.015873 14510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:10:25.016716 14525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.016902 14510 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:25.017014 14510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "05e41ecd2c7045cf8ee6e171c568489f"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-wd30"
server_key: "06e522be9fda5f7f0b4e107360c0c8d9"
server_key_iv: "896517d6d15b7b75522844025f42dc2b"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.017104 14510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:25.029403 14510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:25.031483 14510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:25.031664 14510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:25.035908 14510 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:42193
I20260620 19:10:25.035940 14577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:42193 every 8 connection(s)
I20260620 19:10:25.036338 14510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:25.037375 14578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:25.039696 14578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f: Bootstrap starting.
I20260620 19:10:25.040267 14578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:25.040501 14578 log.cc:824] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:25.041172 14578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f: No bootstrap required, opened a new log
I20260620 19:10:25.042691 14578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e41ecd2c7045cf8ee6e171c568489f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42193 } }
I20260620 19:10:25.042833 14578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:25.042877 14578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e41ecd2c7045cf8ee6e171c568489f, State: Initialized, Role: FOLLOWER
I20260620 19:10:25.043040 14578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e41ecd2c7045cf8ee6e171c568489f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42193 } }
I20260620 19:10:25.043159 14578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:25.043206 14578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:25.043265 14578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:25.043303 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 14510
I20260620 19:10:25.043416 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:10:25.043618 812 external_mini_cluster.cc:1428] Setting key 2ccf0894b5f0755521643a594aeae2f3
I20260620 19:10:25.044442 14578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e41ecd2c7045cf8ee6e171c568489f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42193 } }
I20260620 19:10:25.044531 14578 leader_election.cc:302] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05e41ecd2c7045cf8ee6e171c568489f; no voters:
I20260620 19:10:25.044700 14578 leader_election.cc:288] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:25.044720 14583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:25.044885 14583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 1 LEADER]: Becoming Leader. State: Replica: 05e41ecd2c7045cf8ee6e171c568489f, State: Running, Role: LEADER
I20260620 19:10:25.045020 14583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e41ecd2c7045cf8ee6e171c568489f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42193 } }
I20260620 19:10:25.045022 14578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:25.046315 14584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05e41ecd2c7045cf8ee6e171c568489f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e41ecd2c7045cf8ee6e171c568489f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42193 } } }
I20260620 19:10:25.046447 14584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:25.046314 14585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05e41ecd2c7045cf8ee6e171c568489f. Latest consensus state: current_term: 1 leader_uuid: "05e41ecd2c7045cf8ee6e171c568489f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e41ecd2c7045cf8ee6e171c568489f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42193 } } }
I20260620 19:10:25.046655 14596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:25.046665 14585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:25.047221 14596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:25.049716 14596 catalog_manager.cc:1347] Generated new cluster ID: 74c3d6c7ccca4e0287528c4634351a84
I20260620 19:10:25.049772 14596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:10:25 dist-test-slave-wd30 krb5kdc[14497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:25.054253 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.044153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37896 (local address 127.0.203.62:42193)
0620 19:10:25.044334 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.044343 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.044382 (+ 39us) server_negotiation.cc:407] Connection header received
0620 19:10:25.044478 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.044484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.044557 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.044702 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:25.045421 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.046985 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.047834 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.048373 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.048504 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.051192 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:25.051195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:25.051196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:25.051208 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:25.052771 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.053139 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.053143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.053143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.053239 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.053458 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.053459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.053459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.053688 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:25.053856 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.053951 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.054093 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":105,"threads_started":1}
I20260620 19:10:25.063985 14596 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:25.064535 14596 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:25.076185 14596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f: Generated new TSK 0
I20260620 19:10:25.076416 14596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
I20260620 19:10:25.107643 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:42193
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:25.198416 14604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:25.198632 14604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:25.198659 14604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:25.199841 14604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:25.199884 14604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:25.199911 14604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:10:25.201328 14604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:25.201520 14604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:25.201712 14604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:25.206928 14609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:25.206920 14612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:25.206920 14610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:25.207231 14604 server_base.cc:1034] running on GCE node
I20260620 19:10:25.207495 14604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:25.207785 14604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:25.208926 14604 hybrid_clock.cc:648] HybridClock initialized: now 1781982625208907 us; error 40 us; skew 500 ppm
Jun 20 19:10:25 dist-test-slave-wd30 krb5kdc[14497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:25.211378 14604 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:10:25.211822 14604 webserver.cc:460] Webserver started at http://127.0.203.1:35737/ using document root <none> and password file <none>
I20260620 19:10:25.212052 14604 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:25.212100 14604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:25.212252 14604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:25.213400 14604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "4ac0bbcccaec4bc0bcb934cd315bc756"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-wd30"
server_key: "ddcf783531984b4816fb42342d335223"
server_key_iv: "abf841ec75990b4d4a67e554b61b9cad"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.213764 14604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "4ac0bbcccaec4bc0bcb934cd315bc756"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-wd30"
server_key: "ddcf783531984b4816fb42342d335223"
server_key_iv: "abf841ec75990b4d4a67e554b61b9cad"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.215109 14604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:10:25.215917 14619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.216101 14604 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:10:25.216192 14604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "4ac0bbcccaec4bc0bcb934cd315bc756"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-wd30"
server_key: "ddcf783531984b4816fb42342d335223"
server_key_iv: "abf841ec75990b4d4a67e554b61b9cad"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.216250 14604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:25.235193 14604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:25.237814 14604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:25.238044 14604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:25.238412 14604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:25.238891 14604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:25.238948 14604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.238991 14604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:25.239023 14604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.244995 14604 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:35085
I20260620 19:10:25.245069 14732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:35085 every 8 connection(s)
I20260620 19:10:25.245424 14604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:10:25 dist-test-slave-wd30 krb5kdc[14497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:25.253436 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 14604
I20260620 19:10:25.253523 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:10:25.253791 812 external_mini_cluster.cc:1428] Setting key f7e5521f1bb261623cd1681e07197809
I20260620 19:10:25.255337 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.246187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:44425 (local address 127.0.203.62:42193)
0620 19:10:25.246260 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.246263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.246496 (+ 233us) server_negotiation.cc:407] Connection header received
0620 19:10:25.247146 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.247148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.247165 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.247218 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:25.248205 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.248718 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.249750 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.249966 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.250052 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.252296 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:25.252297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:25.252298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:25.252305 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:25.253710 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.254238 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.254239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.254239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.254308 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.254616 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.254617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.254618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.254732 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:25.254811 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.255191 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.255274 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 19:10:25.255666 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.246179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.1:44425)
0620 19:10:25.246377 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:25.246415 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:25.247009 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:25.247304 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:25.247308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:25.247899 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:25.248097 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.248102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.248830 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.248831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.249650 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.249653 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.250063 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.250064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.250191 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.250699 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:25.250708 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:25.252202 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:25.253793 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.253800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.253803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.254134 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.254383 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.254383 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.254383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.254539 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.254824 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:25.254831 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:25.255005 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 19:10:25.255183 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 19:10:25.255381 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":98,"threads_started":1}
I20260620 19:10:25.256227 14733 heartbeater.cc:347] Connected to a master server at 127.0.203.62:42193
I20260620 19:10:25.256347 14733 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:25.256554 14733 heartbeater.cc:511] Master 127.0.203.62:42193 requested a full tablet report, sending...
I20260620 19:10:25.257093 14542 ts_manager.cc:194] Registered new tserver with Master: 4ac0bbcccaec4bc0bcb934cd315bc756 (127.0.203.1:35085)
I20260620 19:10:25.257776 14542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:44425
WARNING: no policy specified for kudu/127.0.203.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
I20260620 19:10:25.278033 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:42193
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:25.360805 14738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:25.361037 14738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:25.361079 14738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:25.362254 14738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:25.362316 14738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:25.362362 14738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:10:25.363889 14738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:25.364156 14738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:25.364383 14738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:25.369436 14744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:25.369427 14743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:25.369405 14746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:25.369661 14738 server_base.cc:1034] running on GCE node
I20260620 19:10:25.369890 14738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:25.370172 14738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:25.371337 14738 hybrid_clock.cc:648] HybridClock initialized: now 1781982625371316 us; error 34 us; skew 500 ppm
Jun 20 19:10:25 dist-test-slave-wd30 krb5kdc[14497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:25.373977 14738 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:10:25.374413 14738 webserver.cc:460] Webserver started at http://127.0.203.2:33551/ using document root <none> and password file <none>
I20260620 19:10:25.374655 14738 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:25.374720 14738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:25.374862 14738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:25.375998 14738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "5fc6abacdf114f00b6004264326ac657"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-wd30"
server_key: "68269f0ae8f81cd6017c3c62694896b0"
server_key_iv: "97ca673552e3f2e8b70879a222ef89c2"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.376348 14738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "5fc6abacdf114f00b6004264326ac657"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-wd30"
server_key: "68269f0ae8f81cd6017c3c62694896b0"
server_key_iv: "97ca673552e3f2e8b70879a222ef89c2"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.377719 14738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:10:25.378428 14753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.378607 14738 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:10:25.378727 14738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "5fc6abacdf114f00b6004264326ac657"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-wd30"
server_key: "68269f0ae8f81cd6017c3c62694896b0"
server_key_iv: "97ca673552e3f2e8b70879a222ef89c2"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.378800 14738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:25.398613 14738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:25.400815 14738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:25.400988 14738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:25.401237 14738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:25.401552 14738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:25.401604 14738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.401647 14738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:25.401692 14738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.407829 14738 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:37147
I20260620 19:10:25.407951 14866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:37147 every 8 connection(s)
I20260620 19:10:25.408228 14738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:10:25.413291 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 14738
I20260620 19:10:25.413425 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:10:25.413677 812 external_mini_cluster.cc:1428] Setting key 420cb520c2d236fc2b5616484362bc9a
Jun 20 19:10:25 dist-test-slave-wd30 krb5kdc[14497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:25.417904 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.408946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:46857 (local address 127.0.203.62:42193)
0620 19:10:25.409011 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.409013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.409213 (+ 200us) server_negotiation.cc:407] Connection header received
0620 19:10:25.409713 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.409714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.409724 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.409785 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:25.410723 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.411243 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.412261 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.412459 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.412522 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.414998 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:25.415000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:25.415000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:25.415008 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:25.416460 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.416914 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.416915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.416916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.416975 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.417238 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.417239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.417239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.417357 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:25.417441 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.417719 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.417809 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:10:25.418126 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.408923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.2:46857)
0620 19:10:25.409075 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:25.409118 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:25.409631 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:25.409840 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:25.409844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:25.410429 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:25.410630 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.410636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.411324 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.411325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.412163 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.412165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.412562 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.412563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.412708 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.413253 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:25.413265 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:25.414900 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:25.416576 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.416584 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.416586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.416822 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.417039 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.417040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.417040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.417165 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.417434 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:25.417441 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:25.417590 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 19:10:25.417700 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 19:10:25.417873 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260620 19:10:25.418730 14867 heartbeater.cc:347] Connected to a master server at 127.0.203.62:42193
I20260620 19:10:25.418831 14867 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:25.419055 14867 heartbeater.cc:511] Master 127.0.203.62:42193 requested a full tablet report, sending...
I20260620 19:10:25.419523 14542 ts_manager.cc:194] Registered new tserver with Master: 5fc6abacdf114f00b6004264326ac657 (127.0.203.2:37147)
I20260620 19:10:25.419943 14542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:46857
WARNING: no policy specified for kudu/127.0.203.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
I20260620 19:10:25.439628 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:42193
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:25.525476 14872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:25.525698 14872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:25.525727 14872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:25.526866 14872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:25.526906 14872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:25.526932 14872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:10:25.528458 14872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:25.528658 14872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:25.528846 14872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:25.534053 14880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:25.534053 14878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:25.534049 14877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:25.534134 14872 server_base.cc:1034] running on GCE node
I20260620 19:10:25.534461 14872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:25.534740 14872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:25.535919 14872 hybrid_clock.cc:648] HybridClock initialized: now 1781982625535899 us; error 28 us; skew 500 ppm
Jun 20 19:10:25 dist-test-slave-wd30 krb5kdc[14497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:25.538915 14872 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:10:25.539458 14872 webserver.cc:460] Webserver started at http://127.0.203.3:35733/ using document root <none> and password file <none>
I20260620 19:10:25.539680 14872 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:25.539724 14872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:25.539826 14872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:25.540928 14872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "40c36c517a94406a82f67549435981fb"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-wd30"
server_key: "73f1133ca9543a3ae7fb9b5c6f9f9b61"
server_key_iv: "0ea6d2a128b81f5ec70ef1ff55335b8e"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.541244 14872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "40c36c517a94406a82f67549435981fb"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-wd30"
server_key: "73f1133ca9543a3ae7fb9b5c6f9f9b61"
server_key_iv: "0ea6d2a128b81f5ec70ef1ff55335b8e"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.542678 14872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:10:25.543570 14887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.543752 14872 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.543826 14872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "40c36c517a94406a82f67549435981fb"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-wd30"
server_key: "73f1133ca9543a3ae7fb9b5c6f9f9b61"
server_key_iv: "0ea6d2a128b81f5ec70ef1ff55335b8e"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.543883 14872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:25.565629 14872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:25.567796 14872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:25.567916 14872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:25.568117 14872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:25.568380 14872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:25.568411 14872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.568434 14872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:25.568447 14872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.574576 14872 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:36067
I20260620 19:10:25.574620 15000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:36067 every 8 connection(s)
I20260620 19:10:25.575011 14872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:10:25 dist-test-slave-wd30 krb5kdc[14497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:25.584767 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 14872
I20260620 19:10:25.584867 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:10:25.585081 812 external_mini_cluster.cc:1428] Setting key 59db3916837e1010cdd1b17645b5b14b
I20260620 19:10:25.585273 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.575756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:35237 (local address 127.0.203.62:42193)
0620 19:10:25.575839 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.575841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.576057 (+ 216us) server_negotiation.cc:407] Connection header received
0620 19:10:25.576669 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.576670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.576681 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.576768 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:25.577715 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.578189 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.579329 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.579527 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.579586 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.582094 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:25.582096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:25.582096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:25.582106 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:25.583800 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.584216 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.584217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.584217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.584268 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.584493 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.584493 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.584493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.584615 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:25.584690 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.585054 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.585155 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:10:25.585495 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.575712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.3:35237)
0620 19:10:25.575904 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:25.575959 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:25.576548 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:25.576822 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:25.576826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:25.577397 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:25.577624 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.577629 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.578303 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.578304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.579241 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.579244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.579662 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.579664 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.579826 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.580358 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:25.580373 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:25.581970 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:25.583882 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.583890 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.583892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.584148 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.584323 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.584324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.584324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.584438 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.584684 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:25.584691 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:25.584890 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 19:10:25.585012 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 19:10:25.585237 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":108,"threads_started":1}
I20260620 19:10:25.586016 15001 heartbeater.cc:347] Connected to a master server at 127.0.203.62:42193
I20260620 19:10:25.586140 15001 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:25.586338 15001 heartbeater.cc:511] Master 127.0.203.62:42193 requested a full tablet report, sending...
I20260620 19:10:25.586748 14541 ts_manager.cc:194] Registered new tserver with Master: 40c36c517a94406a82f67549435981fb (127.0.203.3:36067)
I20260620 19:10:25.587213 14541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:35237
I20260620 19:10:25.597261 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:10:25.605189 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.598887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37902 (local address 127.0.203.62:42193)
0620 19:10:25.598951 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.598953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.599053 (+ 100us) server_negotiation.cc:407] Connection header received
0620 19:10:25.599112 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.599113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.599143 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.599203 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:25.599785 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.600273 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.600994 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.601199 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.601271 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.602496 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:25.602498 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:25.602498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:25.602505 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:25.604062 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.604417 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.604418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.604419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.604473 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.604637 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.604638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.604638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.604814 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:25.604894 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.605013 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.605076 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:10:25.607108 14541 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:10:25.608872 14541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:25.615535 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.611268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.0.203.1:35085)
0620 19:10:25.611472 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.611477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.611521 (+ 44us) server_negotiation.cc:407] Connection header received
0620 19:10:25.611561 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.611563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.611582 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.611675 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:25.611856 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.613058 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.614394 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.615071 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.615166 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.615229 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.615400 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.615445 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
I20260620 19:10:25.615550 15016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.611352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:36067 (local address 127.0.0.1:44640)
0620 19:10:25.611771 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:25.611782 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:25.611832 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:25.612261 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:25.612262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:25.612264 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:25.612297 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.612298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.613136 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.613137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.614536 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.614539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.615321 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.615322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.615381 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.615384 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:10:25.615429 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 19:10:25.615458 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":74,"thread_start_us":62,"threads_started":1}
I20260620 19:10:25.615767 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.611292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.0.203.2:37147)
0620 19:10:25.611643 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.611649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.611662 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:10:25.611889 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.611892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.611910 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.611987 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:25.612093 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.613194 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.614627 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.615318 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.615396 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.615445 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.615656 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.615700 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":75,"threads_started":1}
I20260620 19:10:25.615895 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.611599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44640 (local address 127.0.203.3:36067)
0620 19:10:25.611745 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.611749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.611907 (+ 158us) server_negotiation.cc:407] Connection header received
0620 19:10:25.612170 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.612172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.612185 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.612266 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:25.612346 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.613039 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.614642 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.615170 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.615232 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.615289 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.615778 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.615819 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260620 19:10:25.615998 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.611181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:37147 (local address 127.0.0.1:36542)
0620 19:10:25.611445 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:25.611454 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:25.611739 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:25.611993 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:25.611994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:25.611996 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:25.612036 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.612038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.613293 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.613294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.614548 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.614550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.615480 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.615481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.615557 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.615562 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:10:25.615873 (+ 311us) client_negotiation.cc:240] Negotiation successful
0620 19:10:25.615922 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":53,"thread_start_us":91,"threads_started":1}
I20260620 19:10:25.615550 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.611149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:35085 (local address 127.0.0.1:43362)
0620 19:10:25.611342 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:25.611386 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:25.611454 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:25.611681 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:25.611684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:25.611687 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:25.611788 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.611791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.613180 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.613181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.614300 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.614303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.615194 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.615195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.615283 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.615297 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 19:10:25.615370 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 19:10:25.615430 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":88,"thread_start_us":60,"threads_started":1}
I20260620 19:10:25.616663 14935 tablet_service.cc:1467] Processing CreateTablet for tablet c5584d517d7d4ae9983d8b67adfdb90d (DEFAULT_TABLE table=test-table [id=529546a93c624958b8a32b796542630c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:25.616714 14801 tablet_service.cc:1467] Processing CreateTablet for tablet c5584d517d7d4ae9983d8b67adfdb90d (DEFAULT_TABLE table=test-table [id=529546a93c624958b8a32b796542630c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:25.616801 14667 tablet_service.cc:1467] Processing CreateTablet for tablet c5584d517d7d4ae9983d8b67adfdb90d (DEFAULT_TABLE table=test-table [id=529546a93c624958b8a32b796542630c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:25.617046 14801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5584d517d7d4ae9983d8b67adfdb90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:25.617049 14935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5584d517d7d4ae9983d8b67adfdb90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:25.617077 14667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5584d517d7d4ae9983d8b67adfdb90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:25.619973 15018 tablet_bootstrap.cc:492] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657: Bootstrap starting.
I20260620 19:10:25.620571 15019 tablet_bootstrap.cc:492] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756: Bootstrap starting.
I20260620 19:10:25.620713 15018 tablet_bootstrap.cc:654] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:25.621188 15019 tablet_bootstrap.cc:654] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:25.621246 15018 log.cc:824] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:25.621762 15019 log.cc:824] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:25.622059 15020 tablet_bootstrap.cc:492] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb: Bootstrap starting.
I20260620 19:10:25.622170 15018 tablet_bootstrap.cc:492] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657: No bootstrap required, opened a new log
I20260620 19:10:25.622272 15018 ts_tablet_manager.cc:1397] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:25.622485 15019 tablet_bootstrap.cc:492] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756: No bootstrap required, opened a new log
I20260620 19:10:25.622550 15019 ts_tablet_manager.cc:1397] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:10:25.622820 15020 tablet_bootstrap.cc:654] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:25.623298 15020 log.cc:824] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:25.623929 15020 tablet_bootstrap.cc:492] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb: No bootstrap required, opened a new log
I20260620 19:10:25.624004 15020 ts_tablet_manager.cc:1397] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:10:25.624339 15018 raft_consensus.cc:348] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } } peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } }
I20260620 19:10:25.624531 15018 raft_consensus.cc:374] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:25.624576 15018 raft_consensus.cc:729] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc6abacdf114f00b6004264326ac657, State: Initialized, Role: FOLLOWER
I20260620 19:10:25.624503 15019 raft_consensus.cc:348] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } } peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } }
I20260620 19:10:25.624683 15019 raft_consensus.cc:374] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:25.624728 15019 raft_consensus.cc:729] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ac0bbcccaec4bc0bcb934cd315bc756, State: Initialized, Role: FOLLOWER
I20260620 19:10:25.624732 15018 consensus_queue.cc:260] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } } peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } }
I20260620 19:10:25.624881 15019 consensus_queue.cc:260] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } } peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } }
I20260620 19:10:25.625056 15018 ts_tablet_manager.cc:1428] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:25.625125 14867 heartbeater.cc:503] Master 127.0.203.62:42193 was elected leader, sending a full tablet report...
I20260620 19:10:25.625137 15019 ts_tablet_manager.cc:1428] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:25.625248 14733 heartbeater.cc:503] Master 127.0.203.62:42193 was elected leader, sending a full tablet report...
I20260620 19:10:25.626765 15020 raft_consensus.cc:348] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } } peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } }
I20260620 19:10:25.626894 15020 raft_consensus.cc:374] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:25.626915 15020 raft_consensus.cc:729] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c36c517a94406a82f67549435981fb, State: Initialized, Role: FOLLOWER
I20260620 19:10:25.627014 15020 consensus_queue.cc:260] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } } peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } }
I20260620 19:10:25.627300 15020 ts_tablet_manager.cc:1428] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 19:10:25.627329 15001 heartbeater.cc:503] Master 127.0.203.62:42193 was elected leader, sending a full tablet report...
W20260620 19:10:25.658874 14868 tablet.cc:2369] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:10:25.746170 14734 tablet.cc:2369] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:10:25.763411 15025 raft_consensus.cc:482] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:25.763541 15025 raft_consensus.cc:504] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } } peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } }
I20260620 19:10:25.763909 15025 leader_election.cc:288] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c36c517a94406a82f67549435981fb (127.0.203.3:36067), 5fc6abacdf114f00b6004264326ac657 (127.0.203.2:37147)
I20260620 19:10:25.768121 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.764207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:57893 (local address 127.0.203.2:37147)
0620 19:10:25.764296 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.764298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.764390 (+ 92us) server_negotiation.cc:407] Connection header received
0620 19:10:25.764453 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.764455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.764468 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.764547 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:25.764791 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.765550 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.767175 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.767756 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.767833 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.767844 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.768009 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.768047 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 19:10:25.768121 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.764305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:48229 (local address 127.0.203.3:36067)
0620 19:10:25.764349 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.764351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.764388 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:10:25.764473 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.764474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.764485 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.764547 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:25.764794 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.765549 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.767170 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.767755 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.767833 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.767844 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.768009 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.768047 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:10:25.768159 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.764128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:37147 (local address 127.0.203.1:57893)
0620 19:10:25.764314 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:25.764326 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:25.764392 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:25.764581 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:25.764582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:25.764586 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:25.764704 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.764706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.765635 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.765636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.767055 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.767059 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.767843 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.767844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.767929 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.767937 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 19:10:25.767995 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 19:10:25.768051 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":116,"thread_start_us":102,"threads_started":1}
I20260620 19:10:25.768162 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.764128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:36067 (local address 127.0.203.1:48229)
0620 19:10:25.764313 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:25.764326 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:25.764392 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:25.764581 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:25.764582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:25.764586 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:25.764711 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.764713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.765632 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.765633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.767055 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.767059 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.767844 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.767845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.767929 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.767935 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:10:25.768003 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 19:10:25.768051 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":99,"thread_start_us":102,"threads_started":1}
I20260620 19:10:25.768491 14821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5584d517d7d4ae9983d8b67adfdb90d" candidate_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fc6abacdf114f00b6004264326ac657" is_pre_election: true
I20260620 19:10:25.768566 14955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5584d517d7d4ae9983d8b67adfdb90d" candidate_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c36c517a94406a82f67549435981fb" is_pre_election: true
I20260620 19:10:25.768657 14821 raft_consensus.cc:2454] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ac0bbcccaec4bc0bcb934cd315bc756 in term 0.
I20260620 19:10:25.768769 14955 raft_consensus.cc:2454] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ac0bbcccaec4bc0bcb934cd315bc756 in term 0.
I20260620 19:10:25.768882 14622 leader_election.cc:302] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ac0bbcccaec4bc0bcb934cd315bc756, 5fc6abacdf114f00b6004264326ac657; no voters:
I20260620 19:10:25.769018 15025 raft_consensus.cc:2781] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:25.769093 15025 raft_consensus.cc:482] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:25.769119 15025 raft_consensus.cc:3037] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:25.769840 15025 raft_consensus.cc:504] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } } peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } }
I20260620 19:10:25.769982 15025 leader_election.cc:288] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 1 election: Requested vote from peers 40c36c517a94406a82f67549435981fb (127.0.203.3:36067), 5fc6abacdf114f00b6004264326ac657 (127.0.203.2:37147)
I20260620 19:10:25.770145 14821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5584d517d7d4ae9983d8b67adfdb90d" candidate_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fc6abacdf114f00b6004264326ac657"
I20260620 19:10:25.770159 14955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5584d517d7d4ae9983d8b67adfdb90d" candidate_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c36c517a94406a82f67549435981fb"
I20260620 19:10:25.770223 14821 raft_consensus.cc:3037] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:25.770256 14955 raft_consensus.cc:3037] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:25.770922 14821 raft_consensus.cc:2454] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ac0bbcccaec4bc0bcb934cd315bc756 in term 1.
I20260620 19:10:25.770999 14955 raft_consensus.cc:2454] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ac0bbcccaec4bc0bcb934cd315bc756 in term 1.
I20260620 19:10:25.771076 14622 leader_election.cc:302] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ac0bbcccaec4bc0bcb934cd315bc756, 5fc6abacdf114f00b6004264326ac657; no voters:
I20260620 19:10:25.771203 15025 raft_consensus.cc:2781] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:25.771344 15025 raft_consensus.cc:686] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 LEADER]: Becoming Leader. State: Replica: 4ac0bbcccaec4bc0bcb934cd315bc756, State: Running, Role: LEADER
I20260620 19:10:25.771453 15025 consensus_queue.cc:237] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } } peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } }
I20260620 19:10:25.772230 14541 catalog_manager.cc:5495] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ac0bbcccaec4bc0bcb934cd315bc756 (127.0.203.1). New cstate: current_term: 1 leader_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } health_report { overall_health: UNKNOWN } } }
I20260620 19:10:25.795873 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.793498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43364 (local address 127.0.203.1:35085)
0620 19:10:25.793567 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.793569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.793578 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:10:25.793616 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.793616 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.793636 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.793695 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:25.793807 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.794419 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.795150 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.795360 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.795416 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.795544 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:25.795648 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:25.795706 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.795737 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.795755 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:10:25.797695 14821 raft_consensus.cc:1261] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657 [term 1 FOLLOWER]: Refusing update from remote peer 4ac0bbcccaec4bc0bcb934cd315bc756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:25.797701 14955 raft_consensus.cc:1261] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb [term 1 FOLLOWER]: Refusing update from remote peer 4ac0bbcccaec4bc0bcb934cd315bc756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:25.798085 15029 consensus_queue.cc:1035] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:25.798211 15025 consensus_queue.cc:1035] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:25.800557 15032 mvcc.cc:205] Tried to move back new op lower bound from 7299000835264839680 to 7299000835160526848. Current Snapshot: MvccSnapshot[applied={T|T < 7299000835264839680}]
I20260620 19:10:25.802044 15033 mvcc.cc:205] Tried to move back new op lower bound from 7299000835264839680 to 7299000835160526848. Current Snapshot: MvccSnapshot[applied={T|T < 7299000835264839680}]
I20260620 19:10:25.803020 14541 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:25.803140 14541 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:25.803946 14541 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f: Sending DeleteTablet for 3 replicas of tablet c5584d517d7d4ae9983d8b67adfdb90d
I20260620 19:10:25.804291 14667 tablet_service.cc:1514] Processing DeleteTablet for tablet c5584d517d7d4ae9983d8b67adfdb90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:25 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:43362
I20260620 19:10:25.804289 14935 tablet_service.cc:1514] Processing DeleteTablet for tablet c5584d517d7d4ae9983d8b67adfdb90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:25 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:44640
I20260620 19:10:25.804296 14801 tablet_service.cc:1514] Processing DeleteTablet for tablet c5584d517d7d4ae9983d8b67adfdb90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:25 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:36542
I20260620 19:10:25.804560 15045 tablet_replica.cc:330] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657: stopping tablet replica
I20260620 19:10:25.804663 15045 raft_consensus.cc:2229] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:25.804685 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 14604
I20260620 19:10:25.804768 15045 raft_consensus.cc:2258] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:25.805410 15043 tablet_replica.cc:330] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb: stopping tablet replica
I20260620 19:10:25.805531 15043 raft_consensus.cc:2229] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:25.805647 15043 raft_consensus.cc:2258] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:25.806121 15043 ts_tablet_manager.cc:1905] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:25.806167 15045 ts_tablet_manager.cc:1905] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:25.807926 15045 ts_tablet_manager.cc:1918] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:25.807989 15045 log.cc:1196] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/c5584d517d7d4ae9983d8b67adfdb90d
I20260620 19:10:25.808156 15043 ts_tablet_manager.cc:1918] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:25.808226 15043 log.cc:1196] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/c5584d517d7d4ae9983d8b67adfdb90d
I20260620 19:10:25.808266 15045 ts_tablet_manager.cc:1939] T c5584d517d7d4ae9983d8b67adfdb90d P 5fc6abacdf114f00b6004264326ac657: Deleting consensus metadata
I20260620 19:10:25.808508 15043 ts_tablet_manager.cc:1939] T c5584d517d7d4ae9983d8b67adfdb90d P 40c36c517a94406a82f67549435981fb: Deleting consensus metadata
I20260620 19:10:25.808736 14528 catalog_manager.cc:4841] TS 5fc6abacdf114f00b6004264326ac657 (127.0.203.2:37147): tablet c5584d517d7d4ae9983d8b67adfdb90d (table test-table [id=529546a93c624958b8a32b796542630c]) successfully deleted
I20260620 19:10:25.808959 14529 catalog_manager.cc:4841] TS 40c36c517a94406a82f67549435981fb (127.0.203.3:36067): tablet c5584d517d7d4ae9983d8b67adfdb90d (table test-table [id=529546a93c624958b8a32b796542630c]) successfully deleted
W20260620 19:10:25.811832 14529 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.1:35085 (error 108)
I20260620 19:10:25.812331 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.812210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:35085 (local address 127.0.0.1:43366)
0620 19:10:25.812268 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:25.812304 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.1:35085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 19:10:25.812463 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 14738
W20260620 19:10:25.812467 14529 catalog_manager.cc:4584] TS 4ac0bbcccaec4bc0bcb934cd315bc756 (127.0.203.1:35085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c5584d517d7d4ae9983d8b67adfdb90d: Network error: Client connection negotiation failed: client connection to 127.0.203.1:35085: connect: Connection refused (error 111)
I20260620 19:10:25.818148 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 14872
I20260620 19:10:25.823184 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 14510
I20260620 19:10:25.828369 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:42193
--webserver_interface=127.0.203.62
--webserver_port=38331
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:42193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:25.909446 15047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:25.909654 15047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:25.909677 15047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:25.910774 15047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:25.910807 15047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:25.910821 15047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:25.910835 15047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:25.910846 15047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:25.912282 15047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:42193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:42193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=38331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:25.912539 15047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:25.912746 15047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:25.921265 15056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:25.921269 15054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:25.921288 15053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:25.921550 15047 server_base.cc:1034] running on GCE node
I20260620 19:10:25.921826 15047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:25.922158 15047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:25.923319 15047 hybrid_clock.cc:648] HybridClock initialized: now 1781982625923306 us; error 38 us; skew 500 ppm
Jun 20 19:10:25 dist-test-slave-wd30 krb5kdc[14497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:25.932395 15047 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:10:25.932965 15047 webserver.cc:460] Webserver started at http://127.0.203.62:38331/ using document root <none> and password file <none>
I20260620 19:10:25.933203 15047 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:25.933262 15047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:25.934625 15047 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:25.935303 15063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.935482 15047 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:25.935570 15047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "05e41ecd2c7045cf8ee6e171c568489f"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-wd30"
server_key: "06e522be9fda5f7f0b4e107360c0c8d9"
server_key_iv: "896517d6d15b7b75522844025f42dc2b"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.935966 15047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:25.949831 15047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:25.952217 15047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:25.952342 15047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:25.956566 15047 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:42193
I20260620 19:10:25.956641 15115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:42193 every 8 connection(s)
I20260620 19:10:25.957019 15047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:25.958597 15116 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:10:25.959228 15116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f: Bootstrap starting.
I20260620 19:10:25.963335 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 15047
I20260620 19:10:25.963630 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:35085
--local_ip_for_outbound_sockets=127.0.203.1
--tserver_master_addrs=127.0.203.62:42193
--webserver_port=35737
--webserver_interface=127.0.203.1
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:10:25.967530 15116 log.cc:824] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:25.969461 15116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:10:25.969774 15116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f: Bootstrap complete.
I20260620 19:10:25.972170 15116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e41ecd2c7045cf8ee6e171c568489f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42193 } }
I20260620 19:10:25.972426 15116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e41ecd2c7045cf8ee6e171c568489f, State: Initialized, Role: FOLLOWER
I20260620 19:10:25.972626 15116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e41ecd2c7045cf8ee6e171c568489f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42193 } }
I20260620 19:10:25.972718 15116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:25.972777 15116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:25.972826 15116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:10:25.973941 15116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e41ecd2c7045cf8ee6e171c568489f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42193 } }
I20260620 19:10:25.974067 15116 leader_election.cc:302] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05e41ecd2c7045cf8ee6e171c568489f; no voters:
I20260620 19:10:25.974256 15116 leader_election.cc:288] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:10:25.974298 15120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:10:25.974424 15120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 2 LEADER]: Becoming Leader. State: Replica: 05e41ecd2c7045cf8ee6e171c568489f, State: Running, Role: LEADER
I20260620 19:10:25.974550 15116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:25.974568 15120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e41ecd2c7045cf8ee6e171c568489f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42193 } }
I20260620 19:10:25.974953 15121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "05e41ecd2c7045cf8ee6e171c568489f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e41ecd2c7045cf8ee6e171c568489f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42193 } } }
I20260620 19:10:25.974951 15122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05e41ecd2c7045cf8ee6e171c568489f. Latest consensus state: current_term: 2 leader_uuid: "05e41ecd2c7045cf8ee6e171c568489f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e41ecd2c7045cf8ee6e171c568489f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42193 } } }
I20260620 19:10:25.975102 15122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:25.975082 15121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:25.975308 15128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:25.976153 15128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:25.976270 15128 catalog_manager.cc:1259] Loaded cluster ID: 74c3d6c7ccca4e0287528c4634351a84
I20260620 19:10:25.976507 15128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:25.976781 15128 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 19:10:25.979523 15128 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 19:10:25.987949 15128 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:25.988191 15128 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f: Loaded TSK: 0
I20260620 19:10:25.988408 15128 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:10:26.061921 15118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:26.062150 15118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:26.062192 15118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:26.063417 15118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:26.063476 15118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:26.063525 15118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:10:26.065001 15118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:35085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=35737
--tserver_master_addrs=127.0.203.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:26.065261 15118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:26.065490 15118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:26.070475 15146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:26.070408 15143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:26.070688 15144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:26.070825 15118 server_base.cc:1034] running on GCE node
I20260620 19:10:26.071022 15118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:26.071305 15118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:26.072474 15118 hybrid_clock.cc:648] HybridClock initialized: now 1781982626072456 us; error 36 us; skew 500 ppm
Jun 20 19:10:26 dist-test-slave-wd30 krb5kdc[14497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:26.075265 15118 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:10:26.075798 15118 webserver.cc:460] Webserver started at http://127.0.203.1:35737/ using document root <none> and password file <none>
I20260620 19:10:26.076095 15118 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:26.076179 15118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:26.077631 15118 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:10:26.078397 15153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.078593 15118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:26.078701 15118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "4ac0bbcccaec4bc0bcb934cd315bc756"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-wd30"
server_key: "ddcf783531984b4816fb42342d335223"
server_key_iv: "abf841ec75990b4d4a67e554b61b9cad"
server_key_version: "encryptionkey@0"
I20260620 19:10:26.079155 15118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:26.099614 15118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:26.101858 15118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:26.102028 15118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:26.102255 15118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:26.102702 15160 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:10:26.104653 15118 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:10:26.104710 15118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 19:10:26.104766 15118 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:10:26.105374 15118 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:10:26.105425 15118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 19:10:26.105512 15160 tablet_bootstrap.cc:492] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756: Bootstrap starting.
I20260620 19:10:26.112697 15118 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:35085
I20260620 19:10:26.112869 15267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:35085 every 8 connection(s)
I20260620 19:10:26.113152 15118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:10:26.116799 15160 log.cc:824] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:26.119058 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 15118
I20260620 19:10:26.119494 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:37147
--local_ip_for_outbound_sockets=127.0.203.2
--tserver_master_addrs=127.0.203.62:42193
--webserver_port=33551
--webserver_interface=127.0.203.2
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:10:26.119438 15160 tablet_bootstrap.cc:492] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:10:26.120110 15160 tablet_bootstrap.cc:492] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756: Bootstrap complete.
I20260620 19:10:26.121374 15160 ts_tablet_manager.cc:1397] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
Jun 20 19:10:26 dist-test-slave-wd30 krb5kdc[14497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:26.122685 15160 raft_consensus.cc:348] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } } peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } }
I20260620 19:10:26.123003 15160 raft_consensus.cc:729] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ac0bbcccaec4bc0bcb934cd315bc756, State: Initialized, Role: FOLLOWER
I20260620 19:10:26.123183 15160 consensus_queue.cc:260] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } } peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } }
I20260620 19:10:26.123459 15160 ts_tablet_manager.cc:1428] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20260620 19:10:26.126202 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.114008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:41867 (local address 127.0.203.62:42193)
0620 19:10:26.115254 (+ 1246us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.115261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.115292 (+ 31us) server_negotiation.cc:407] Connection header received
0620 19:10:26.115397 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.115401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.115454 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.115590 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.116224 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.117303 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.118445 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.118887 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.119543 (+ 656us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.122235 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.122239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.122240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.122253 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.124064 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.124587 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.124590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.124590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.124716 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.125036 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.125037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.125037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.125417 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.125505 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.125880 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.126022 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1191,"thread_start_us":84,"threads_started":1}
I20260620 19:10:26.126273 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.113905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.1:41867)
0620 19:10:26.114091 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.114142 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.114637 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.115629 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.115633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.115916 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.116108 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.116113 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.117421 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.117422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.118352 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.118355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.119016 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.119017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.119233 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.119860 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.119870 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.122127 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.124153 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.124160 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.124162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.124480 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.124809 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.124810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.124810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.124952 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.125497 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.125504 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.125683 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.125811 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.126008 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
I20260620 19:10:26.126991 15268 heartbeater.cc:347] Connected to a master server at 127.0.203.62:42193
I20260620 19:10:26.127094 15268 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:26.127307 15268 heartbeater.cc:511] Master 127.0.203.62:42193 requested a full tablet report, sending...
I20260620 19:10:26.128122 15080 ts_manager.cc:194] Registered new tserver with Master: 4ac0bbcccaec4bc0bcb934cd315bc756 (127.0.203.1:35085)
I20260620 19:10:26.129407 15080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:41867
I20260620 19:10:26.131034 15268 heartbeater.cc:503] Master 127.0.203.62:42193 was elected leader, sending a full tablet report...
Jun 20 19:10:26 dist-test-slave-wd30 krb5kdc[14497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for kudu/127.0.203.1@KRBTEST.COM
I20260620 19:10:26.138895 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.130285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43380 (local address 127.0.203.1:35085)
0620 19:10:26.130474 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.130479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.130492 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:10:26.130525 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.130526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.130541 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.131252 (+ 711us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.131317 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.132165 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.133495 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.133950 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.134025 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.136085 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.136094 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.136094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.136101 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.137624 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.138032 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.138035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.138035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.138095 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.138305 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.138306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.138307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.138441 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.138530 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.138701 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.138792 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":93,"threads_started":1}
I20260620 19:10:26.138895 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.129861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:35085 (local address 127.0.0.1:43380)
0620 19:10:26.130039 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.130079 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.130148 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.130633 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.130638 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.131000 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.131062 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.131064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.132263 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.132264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.133390 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.133394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.134032 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.134032 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.134110 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.134674 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.134678 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.135996 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.137727 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.137734 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.137736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.137935 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.138163 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.138164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.138164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.138228 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.138519 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.138525 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.138582 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.138665 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.138799 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":101,"threads_started":1}
I20260620 19:10:26.139752 15202 tablet_service.cc:1514] Processing DeleteTablet for tablet c5584d517d7d4ae9983d8b67adfdb90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:25 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:43380
I20260620 19:10:26.140355 15277 tablet_replica.cc:330] stopping tablet replica
I20260620 19:10:26.140479 15277 raft_consensus.cc:2229] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:26.140548 15277 raft_consensus.cc:2258] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:26.140858 15277 ts_tablet_manager.cc:1905] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:26.143158 15277 ts_tablet_manager.cc:1918] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:26.143236 15277 log.cc:1196] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/wal/wals/c5584d517d7d4ae9983d8b67adfdb90d
I20260620 19:10:26.143492 15277 ts_tablet_manager.cc:1939] T c5584d517d7d4ae9983d8b67adfdb90d P 4ac0bbcccaec4bc0bcb934cd315bc756: Deleting consensus metadata
I20260620 19:10:26.144006 15067 catalog_manager.cc:4841] TS 4ac0bbcccaec4bc0bcb934cd315bc756 (127.0.203.1:35085): tablet c5584d517d7d4ae9983d8b67adfdb90d (table test-table [id=529546a93c624958b8a32b796542630c]) successfully deleted
W20260620 19:10:26.209635 15273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:26.209856 15273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:26.209880 15273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:26.210980 15273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:26.211018 15273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:26.211043 15273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:10:26.212507 15273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:37147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=33551
--tserver_master_addrs=127.0.203.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:26.212703 15273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:26.212893 15273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:26.217983 15282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:26.217962 15283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:26.217991 15273 server_base.cc:1034] running on GCE node
W20260620 19:10:26.217943 15285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:26.218338 15273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:26.218613 15273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:26.219766 15273 hybrid_clock.cc:648] HybridClock initialized: now 1781982626219762 us; error 34 us; skew 500 ppm
Jun 20 19:10:26 dist-test-slave-wd30 krb5kdc[14497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:26.222069 15273 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:10:26.222546 15273 webserver.cc:460] Webserver started at http://127.0.203.2:33551/ using document root <none> and password file <none>
I20260620 19:10:26.222765 15273 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:26.222842 15273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:26.224206 15273 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:26.224802 15292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.224983 15273 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:10:26.225085 15273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "5fc6abacdf114f00b6004264326ac657"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-wd30"
server_key: "68269f0ae8f81cd6017c3c62694896b0"
server_key_iv: "97ca673552e3f2e8b70879a222ef89c2"
server_key_version: "encryptionkey@0"
I20260620 19:10:26.225440 15273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:26.246371 15273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:26.248596 15273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:26.248807 15273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:26.249053 15273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:26.249372 15273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:26.249424 15273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.249459 15273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:26.249487 15273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.255441 15273 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:37147
I20260620 19:10:26.255486 15405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:37147 every 8 connection(s)
I20260620 19:10:26.255831 15273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:10:26.257180 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 15273
I20260620 19:10:26.257468 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:36067
--local_ip_for_outbound_sockets=127.0.203.3
--tserver_master_addrs=127.0.203.62:42193
--webserver_port=35733
--webserver_interface=127.0.203.3
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:10:26 dist-test-slave-wd30 krb5kdc[14497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:26.265822 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.256510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.2:59181)
0620 19:10:26.256677 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.256713 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.257205 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.257456 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.257459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.258118 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.258321 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.258326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.259069 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.259075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.259921 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.259923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.260338 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.260340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.260463 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.261094 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.261102 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.262713 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.264326 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.264333 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.264336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.264568 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.264771 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.264772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.264772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.264907 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.265303 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.265308 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.265436 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.265525 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.265651 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260620 19:10:26.265826 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.256504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:59181 (local address 127.0.203.62:42193)
0620 19:10:26.256577 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.256579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.256800 (+ 221us) server_negotiation.cc:407] Connection header received
0620 19:10:26.257306 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.257308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.257324 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.257381 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.258452 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.258964 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.260012 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.260241 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.260328 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.262808 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.262810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.262810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.262819 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.264244 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.264651 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.264652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.264653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.264709 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.265038 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.265040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.265040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.265223 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.265315 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.265573 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.265685 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:10:26.266503 15406 heartbeater.cc:347] Connected to a master server at 127.0.203.62:42193
I20260620 19:10:26.266597 15406 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:26.266789 15406 heartbeater.cc:511] Master 127.0.203.62:42193 requested a full tablet report, sending...
I20260620 19:10:26.267225 15080 ts_manager.cc:194] Registered new tserver with Master: 5fc6abacdf114f00b6004264326ac657 (127.0.203.2:37147)
I20260620 19:10:26.267604 15080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:59181
W20260620 19:10:26.341905 15409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:26.342116 15409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:26.342141 15409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:26.343351 15409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:26.343393 15409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:26.343443 15409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:10:26.344944 15409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:36067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=35733
--tserver_master_addrs=127.0.203.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:26.345180 15409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:26.345405 15409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:26.350564 15417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:26.350571 15415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:26.350618 15409 server_base.cc:1034] running on GCE node
W20260620 19:10:26.350576 15414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:26.350873 15409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:26.351167 15409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:26.352341 15409 hybrid_clock.cc:648] HybridClock initialized: now 1781982626352332 us; error 33 us; skew 500 ppm
Jun 20 19:10:26 dist-test-slave-wd30 krb5kdc[14497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:26.354817 15409 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:10:26.355319 15409 webserver.cc:460] Webserver started at http://127.0.203.3:35733/ using document root <none> and password file <none>
I20260620 19:10:26.355506 15409 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:26.355557 15409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:26.356830 15409 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:26.357373 15424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.357561 15409 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:10:26.357632 15409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "40c36c517a94406a82f67549435981fb"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-wd30"
server_key: "73f1133ca9543a3ae7fb9b5c6f9f9b61"
server_key_iv: "0ea6d2a128b81f5ec70ef1ff55335b8e"
server_key_version: "encryptionkey@0"
I20260620 19:10:26.357949 15409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:26.370958 15409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:26.373392 15409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:26.373507 15409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:26.373703 15409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:26.373948 15409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:26.373979 15409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.374001 15409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:26.374014 15409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.380074 15409 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:36067
I20260620 19:10:26.380115 15537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:36067 every 8 connection(s)
I20260620 19:10:26.380471 15409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:26.382347 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 15409
Jun 20 19:10:26 dist-test-slave-wd30 krb5kdc[14497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:26.389406 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.382773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37916 (local address 127.0.203.62:42193)
0620 19:10:26.382924 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.382926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.382935 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:10:26.382983 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.382984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.382997 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.383064 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.383654 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.384132 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.384860 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.385070 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.385150 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.386411 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.386412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.386413 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.386421 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.388219 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.388552 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.388553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.388554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.388614 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.388806 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.388807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.388807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.388935 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.389133 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.389169 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.389255 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260620 19:10:26.390429 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.381190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:54683 (local address 127.0.203.62:42193)
0620 19:10:26.381237 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.381239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.381480 (+ 241us) server_negotiation.cc:407] Connection header received
0620 19:10:26.382033 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.382034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.382047 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.382107 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.383374 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.383865 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.384848 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.385040 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.385120 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.387553 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.387555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.387555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.387562 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.388915 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.389354 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.389355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.389355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.389415 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.389727 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.389728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.389728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.389882 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.389944 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.390293 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.390365 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:10:26.390695 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.381184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.3:54683)
0620 19:10:26.381353 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.381395 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.381930 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.382210 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.382214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.383028 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.383281 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.383286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.383954 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.383955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.384758 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.384760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.385130 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.385131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.385253 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.385883 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.385891 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.387444 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.389021 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.389028 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.389031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.389260 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.389479 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.389480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.389481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.389619 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.389956 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.389962 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.390147 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.390275 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.390437 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20260620 19:10:26.391139 15538 heartbeater.cc:347] Connected to a master server at 127.0.203.62:42193
I20260620 19:10:26.391245 15538 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:26.391430 15538 heartbeater.cc:511] Master 127.0.203.62:42193 requested a full tablet report, sending...
I20260620 19:10:26.391772 15080 ts_manager.cc:194] Registered new tserver with Master: 40c36c517a94406a82f67549435981fb (127.0.203.3:36067)
I20260620 19:10:26.392144 15080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:54683
I20260620 19:10:26.401172 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:10:26.404148 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.401572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37924 (local address 127.0.203.62:42193)
0620 19:10:26.401633 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.401635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.401720 (+ 85us) server_negotiation.cc:407] Connection header received
0620 19:10:26.401799 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.401800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.401837 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.401909 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:26.402096 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.402589 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.403459 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.403646 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.403701 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.403861 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:26.403975 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:26.404038 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.404071 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.404088 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:10:26.404486 15080 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:10:26.406304 15080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:26.408809 15202 tablet_service.cc:1467] Processing CreateTablet for tablet 8351285b2f2e46c491d78c5bb45873a4 (DEFAULT_TABLE table=test-table [id=88497599a5bb40c9b88b5bbb236de57c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:26.409039 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8351285b2f2e46c491d78c5bb45873a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:26.410550 15160 tablet_bootstrap.cc:492] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756: Bootstrap starting.
I20260620 19:10:26.410888 15160 tablet_bootstrap.cc:654] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:26.411887 15160 tablet_bootstrap.cc:492] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756: No bootstrap required, opened a new log
I20260620 19:10:26.411948 15160 ts_tablet_manager.cc:1397] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:26.412103 15160 raft_consensus.cc:348] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } } peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } }
I20260620 19:10:26.412189 15160 raft_consensus.cc:374] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:26.412237 15160 raft_consensus.cc:729] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ac0bbcccaec4bc0bcb934cd315bc756, State: Initialized, Role: FOLLOWER
I20260620 19:10:26.412292 15160 consensus_queue.cc:260] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } } peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } }
I20260620 19:10:26.412401 15160 ts_tablet_manager.cc:1428] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.412461 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.408327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:36067 (local address 127.0.0.1:44646)
0620 19:10:26.408438 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.408452 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.408506 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.408867 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.408868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.408869 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:26.408920 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.408922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.410177 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.410178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.411227 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.411229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.412148 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.412150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.412237 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.412266 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.412320 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.412382 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 19:10:26.413017 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.408541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44646 (local address 127.0.203.3:36067)
0620 19:10:26.408692 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.408701 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.408712 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:10:26.408752 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.408754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.408768 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.408863 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:26.408991 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.410080 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.411303 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.412063 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.412144 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.412209 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.412880 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.412933 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260620 19:10:26.413812 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 8351285b2f2e46c491d78c5bb45873a4 (DEFAULT_TABLE table=test-table [id=88497599a5bb40c9b88b5bbb236de57c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:26.414206 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8351285b2f2e46c491d78c5bb45873a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:26.414836 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.408352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:37147 (local address 127.0.0.1:36546)
0620 19:10:26.408541 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.408551 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.408595 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.411005 (+ 2410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.411006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.411008 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:26.411046 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.411047 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.412339 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.412340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.413520 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.413524 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.414571 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.414572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.414649 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.414654 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.414715 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.414760 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":112,"threads_started":1}
I20260620 19:10:26.414852 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.408456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.0.203.2:37147)
0620 19:10:26.410749 (+ 2293us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.410754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.410767 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:10:26.410811 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.410813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.410831 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.410925 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:26.411197 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.412215 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.413647 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.414464 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.414566 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.414638 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.414730 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.414769 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2147,"thread_start_us":87,"threads_started":1}
I20260620 19:10:26.415514 15340 tablet_service.cc:1467] Processing CreateTablet for tablet 8351285b2f2e46c491d78c5bb45873a4 (DEFAULT_TABLE table=test-table [id=88497599a5bb40c9b88b5bbb236de57c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:26.415839 15340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8351285b2f2e46c491d78c5bb45873a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:26.417452 15548 tablet_bootstrap.cc:492] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb: Bootstrap starting.
I20260620 19:10:26.418095 15548 tablet_bootstrap.cc:654] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:26.418632 15548 log.cc:824] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:26.418905 15549 tablet_bootstrap.cc:492] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657: Bootstrap starting.
I20260620 19:10:26.419624 15549 tablet_bootstrap.cc:654] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:26.419740 15548 tablet_bootstrap.cc:492] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb: No bootstrap required, opened a new log
I20260620 19:10:26.419812 15548 ts_tablet_manager.cc:1397] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:26.420269 15549 log.cc:824] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:26.421008 15549 tablet_bootstrap.cc:492] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657: No bootstrap required, opened a new log
I20260620 19:10:26.421092 15549 ts_tablet_manager.cc:1397] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:26.421336 15548 raft_consensus.cc:348] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } } peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } }
I20260620 19:10:26.421504 15548 raft_consensus.cc:374] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:26.421527 15548 raft_consensus.cc:729] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c36c517a94406a82f67549435981fb, State: Initialized, Role: FOLLOWER
I20260620 19:10:26.421644 15548 consensus_queue.cc:260] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } } peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } }
I20260620 19:10:26.421905 15548 ts_tablet_manager.cc:1428] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:26.421962 15538 heartbeater.cc:503] Master 127.0.203.62:42193 was elected leader, sending a full tablet report...
I20260620 19:10:26.423223 15549 raft_consensus.cc:348] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } } peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } }
I20260620 19:10:26.423396 15549 raft_consensus.cc:374] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:26.423441 15549 raft_consensus.cc:729] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc6abacdf114f00b6004264326ac657, State: Initialized, Role: FOLLOWER
I20260620 19:10:26.423565 15549 consensus_queue.cc:260] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } } peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } }
I20260620 19:10:26.423827 15549 ts_tablet_manager.cc:1428] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:26.423880 15406 heartbeater.cc:503] Master 127.0.203.62:42193 was elected leader, sending a full tablet report...
W20260620 19:10:26.506379 15407 tablet.cc:2369] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:10:26.614121 15269 tablet.cc:2369] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:10:26.631057 15539 tablet.cc:2369] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:10:26.728487 15274 raft_consensus.cc:482] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:26.728772 15274 raft_consensus.cc:504] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } } peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } }
I20260620 19:10:26.729197 15274 leader_election.cc:288] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c36c517a94406a82f67549435981fb (127.0.203.3:36067), 5fc6abacdf114f00b6004264326ac657 (127.0.203.2:37147)
I20260620 19:10:26.732201 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.729474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:57385 (local address 127.0.203.3:36067)
0620 19:10:26.729533 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.729535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.729619 (+ 84us) server_negotiation.cc:407] Connection header received
0620 19:10:26.729721 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.729722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.729734 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.729803 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:26.729989 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.730435 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.731446 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.731829 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.731885 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.731894 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.732079 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.732123 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 19:10:26.732210 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.729623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:44113 (local address 127.0.203.2:37147)
0620 19:10:26.729670 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.729671 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.729679 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:10:26.729711 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.729712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.729724 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.729795 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:26.729935 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.730428 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.731480 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.731852 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.731939 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.731949 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.732105 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.732147 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:10:26.732213 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.729382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:36067 (local address 127.0.203.1:57385)
0620 19:10:26.729537 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.729552 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.729634 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.729817 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.729818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.729821 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:26.729916 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.729918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.730528 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.730529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.731365 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.731367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.731906 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.731907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.731978 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.732007 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.732050 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.732120 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":11,"thread_start_us":85,"threads_started":1}
I20260620 19:10:26.732229 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.729412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:37147 (local address 127.0.203.1:44113)
0620 19:10:26.729561 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.729581 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.729643 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.729794 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.729795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.729797 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:26.729860 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.729863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.730504 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.730505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.731372 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.731374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.731955 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.731956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.732000 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.732016 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.732086 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.732135 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":44,"thread_start_us":79,"threads_started":1}
I20260620 19:10:26.732695 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8351285b2f2e46c491d78c5bb45873a4" candidate_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c36c517a94406a82f67549435981fb" is_pre_election: true
I20260620 19:10:26.732836 15492 raft_consensus.cc:2454] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ac0bbcccaec4bc0bcb934cd315bc756 in term 0.
I20260620 19:10:26.733049 15157 leader_election.cc:302] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c36c517a94406a82f67549435981fb, 4ac0bbcccaec4bc0bcb934cd315bc756; no voters:
I20260620 19:10:26.733053 15360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8351285b2f2e46c491d78c5bb45873a4" candidate_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fc6abacdf114f00b6004264326ac657" is_pre_election: true
I20260620 19:10:26.733198 15360 raft_consensus.cc:2454] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ac0bbcccaec4bc0bcb934cd315bc756 in term 0.
I20260620 19:10:26.733188 15274 raft_consensus.cc:2781] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:26.733251 15274 raft_consensus.cc:482] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:26.733289 15274 raft_consensus.cc:3037] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:26.733978 15274 raft_consensus.cc:504] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } } peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } }
I20260620 19:10:26.734136 15274 leader_election.cc:288] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 1 election: Requested vote from peers 40c36c517a94406a82f67549435981fb (127.0.203.3:36067), 5fc6abacdf114f00b6004264326ac657 (127.0.203.2:37147)
I20260620 19:10:26.734267 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8351285b2f2e46c491d78c5bb45873a4" candidate_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c36c517a94406a82f67549435981fb"
I20260620 19:10:26.734304 15360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8351285b2f2e46c491d78c5bb45873a4" candidate_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fc6abacdf114f00b6004264326ac657"
I20260620 19:10:26.734354 15492 raft_consensus.cc:3037] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:26.734406 15360 raft_consensus.cc:3037] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:26.735009 15492 raft_consensus.cc:2454] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ac0bbcccaec4bc0bcb934cd315bc756 in term 1.
I20260620 19:10:26.735179 15360 raft_consensus.cc:2454] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ac0bbcccaec4bc0bcb934cd315bc756 in term 1.
I20260620 19:10:26.735203 15157 leader_election.cc:302] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c36c517a94406a82f67549435981fb, 4ac0bbcccaec4bc0bcb934cd315bc756; no voters:
I20260620 19:10:26.735289 15274 raft_consensus.cc:2781] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:26.735412 15274 raft_consensus.cc:686] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 LEADER]: Becoming Leader. State: Replica: 4ac0bbcccaec4bc0bcb934cd315bc756, State: Running, Role: LEADER
I20260620 19:10:26.735502 15274 consensus_queue.cc:237] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } } peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } }
I20260620 19:10:26.736253 15080 catalog_manager.cc:5495] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ac0bbcccaec4bc0bcb934cd315bc756 (127.0.203.1). New cstate: current_term: 1 leader_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } health_report { overall_health: HEALTHY } } }
I20260620 19:10:26.765537 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.763248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46504 (local address 127.0.203.1:35085)
0620 19:10:26.763438 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.763440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.763448 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:10:26.763486 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.763487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.763508 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.763581 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:26.763687 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.764178 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.764845 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.765041 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.765094 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.765228 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:26.765323 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:26.765376 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.765410 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.765427 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":108,"threads_started":1}
I20260620 19:10:26.767357 15492 raft_consensus.cc:1261] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb [term 1 FOLLOWER]: Refusing update from remote peer 4ac0bbcccaec4bc0bcb934cd315bc756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:26.767359 15360 raft_consensus.cc:1261] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657 [term 1 FOLLOWER]: Refusing update from remote peer 4ac0bbcccaec4bc0bcb934cd315bc756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:26.767704 15556 consensus_queue.cc:1035] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:26.767839 15274 consensus_queue.cc:1035] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:26.770215 15559 mvcc.cc:205] Tried to move back new op lower bound from 7299000839236808704 to 7299000839109513216. Current Snapshot: MvccSnapshot[applied={T|T < 7299000839236808704}]
I20260620 19:10:26.770336 15561 mvcc.cc:205] Tried to move back new op lower bound from 7299000839236808704 to 7299000839109513216. Current Snapshot: MvccSnapshot[applied={T|T < 7299000839236808704}]
I20260620 19:10:26.770424 15560 mvcc.cc:205] Tried to move back new op lower bound from 7299000839236808704 to 7299000839109513216. Current Snapshot: MvccSnapshot[applied={T|T < 7299000839236808704}]
I20260620 19:10:26.772935 15079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:26.773044 15079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:26.773809 15079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f: Sending DeleteTablet for 3 replicas of tablet 8351285b2f2e46c491d78c5bb45873a4
I20260620 19:10:26.774138 15472 tablet_service.cc:1514] Processing DeleteTablet for tablet 8351285b2f2e46c491d78c5bb45873a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:26 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:44646
I20260620 19:10:26.774152 15340 tablet_service.cc:1514] Processing DeleteTablet for tablet 8351285b2f2e46c491d78c5bb45873a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:26 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:36546
I20260620 19:10:26.774188 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 15118
I20260620 19:10:26.774926 15570 tablet_replica.cc:330] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657: stopping tablet replica
I20260620 19:10:26.775033 15570 raft_consensus.cc:2229] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:26.775167 15570 raft_consensus.cc:2258] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:26.775321 15569 tablet_replica.cc:330] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb: stopping tablet replica
I20260620 19:10:26.775420 15569 raft_consensus.cc:2229] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:26.775521 15569 raft_consensus.cc:2258] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:26.776007 15570 ts_tablet_manager.cc:1905] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:26.778131 15570 ts_tablet_manager.cc:1918] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:26.778215 15570 log.cc:1196] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/8351285b2f2e46c491d78c5bb45873a4
I20260620 19:10:26.778514 15570 ts_tablet_manager.cc:1939] T 8351285b2f2e46c491d78c5bb45873a4 P 5fc6abacdf114f00b6004264326ac657: Deleting consensus metadata
I20260620 19:10:26.778934 15066 catalog_manager.cc:4841] TS 5fc6abacdf114f00b6004264326ac657 (127.0.203.2:37147): tablet 8351285b2f2e46c491d78c5bb45873a4 (table test-table [id=88497599a5bb40c9b88b5bbb236de57c]) successfully deleted
W20260620 19:10:26.780939 15067 connection.cc:535] client connection to 127.0.203.1:35085 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 19:10:26.781054 15067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:10:26.781101 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 15273
I20260620 19:10:26.782088 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.781935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:35085 (local address 127.0.0.1:46516)
0620 19:10:26.781998 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.782045 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.1:35085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 19:10:26.782166 15569 ts_tablet_manager.cc:1905] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:10:26.782218 15067 catalog_manager.cc:4584] TS 4ac0bbcccaec4bc0bcb934cd315bc756 (127.0.203.1:35085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8351285b2f2e46c491d78c5bb45873a4: Network error: Client connection negotiation failed: client connection to 127.0.203.1:35085: connect: Connection refused (error 111)
I20260620 19:10:26.783629 15569 ts_tablet_manager.cc:1918] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:26.783699 15569 log.cc:1196] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/8351285b2f2e46c491d78c5bb45873a4
I20260620 19:10:26.783957 15569 ts_tablet_manager.cc:1939] T 8351285b2f2e46c491d78c5bb45873a4 P 40c36c517a94406a82f67549435981fb: Deleting consensus metadata
I20260620 19:10:26.784346 15067 catalog_manager.cc:4841] TS 40c36c517a94406a82f67549435981fb (127.0.203.3:36067): tablet 8351285b2f2e46c491d78c5bb45873a4 (table test-table [id=88497599a5bb40c9b88b5bbb236de57c]) successfully deleted
I20260620 19:10:26.786916 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 15409
I20260620 19:10:26.791483 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 15047
I20260620 19:10:26.796962 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:42193
--webserver_interface=127.0.203.62
--webserver_port=38331
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:42193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:26.881534 15572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:26.881742 15572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:26.881765 15572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:26.883021 15572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:26.883069 15572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:26.883086 15572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:26.883101 15572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:26.883183 15572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:26.884589 15572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:42193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:42193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=38331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:26.884835 15572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:26.885080 15572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:26.889794 15577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:26.890038 15580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:26.890043 15578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:26.890168 15572 server_base.cc:1034] running on GCE node
I20260620 19:10:26.890398 15572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:26.890700 15572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:26.891853 15572 hybrid_clock.cc:648] HybridClock initialized: now 1781982626891839 us; error 31 us; skew 500 ppm
Jun 20 19:10:26 dist-test-slave-wd30 krb5kdc[14497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:26.894203 15572 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:10:26.894626 15572 webserver.cc:460] Webserver started at http://127.0.203.62:38331/ using document root <none> and password file <none>
I20260620 19:10:26.894847 15572 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:26.894893 15572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:26.896267 15572 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:26.896951 15587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.897138 15572 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:26.897221 15572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "05e41ecd2c7045cf8ee6e171c568489f"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-wd30"
server_key: "06e522be9fda5f7f0b4e107360c0c8d9"
server_key_iv: "896517d6d15b7b75522844025f42dc2b"
server_key_version: "encryptionkey@0"
I20260620 19:10:26.897570 15572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:26.920603 15572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:26.922780 15572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:26.922892 15572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:26.926985 15572 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:42193
I20260620 19:10:26.927062 15639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:42193 every 8 connection(s)
I20260620 19:10:26.927471 15572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:26.929116 15640 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:10:26.929796 15640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f: Bootstrap starting.
I20260620 19:10:26.932276 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 15572
I20260620 19:10:26.932582 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:35085
--local_ip_for_outbound_sockets=127.0.203.1
--tserver_master_addrs=127.0.203.62:42193
--webserver_port=35737
--webserver_interface=127.0.203.1
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:10:26.938262 15640 log.cc:824] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:26.940636 15640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:10:26.940958 15640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f: Bootstrap complete.
I20260620 19:10:26.943297 15640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e41ecd2c7045cf8ee6e171c568489f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42193 } }
I20260620 19:10:26.943547 15640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e41ecd2c7045cf8ee6e171c568489f, State: Initialized, Role: FOLLOWER
I20260620 19:10:26.943722 15640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e41ecd2c7045cf8ee6e171c568489f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42193 } }
I20260620 19:10:26.943797 15640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:26.943849 15640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:26.943897 15640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:10:26.945004 15640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e41ecd2c7045cf8ee6e171c568489f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42193 } }
I20260620 19:10:26.945123 15640 leader_election.cc:302] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05e41ecd2c7045cf8ee6e171c568489f; no voters:
I20260620 19:10:26.945315 15640 leader_election.cc:288] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 19:10:26.945354 15644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:10:26.945500 15644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [term 3 LEADER]: Becoming Leader. State: Replica: 05e41ecd2c7045cf8ee6e171c568489f, State: Running, Role: LEADER
I20260620 19:10:26.945596 15644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e41ecd2c7045cf8ee6e171c568489f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42193 } }
I20260620 19:10:26.945596 15640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:26.945900 15646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05e41ecd2c7045cf8ee6e171c568489f. Latest consensus state: current_term: 3 leader_uuid: "05e41ecd2c7045cf8ee6e171c568489f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e41ecd2c7045cf8ee6e171c568489f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42193 } } }
I20260620 19:10:26.945997 15646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:26.945899 15645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "05e41ecd2c7045cf8ee6e171c568489f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05e41ecd2c7045cf8ee6e171c568489f" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42193 } } }
I20260620 19:10:26.946205 15645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05e41ecd2c7045cf8ee6e171c568489f [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:26.946312 15648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:26.947165 15648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:26.947278 15648 catalog_manager.cc:1259] Loaded cluster ID: 74c3d6c7ccca4e0287528c4634351a84
I20260620 19:10:26.947321 15648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 19:10:26.947569 15648 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9c75fbb9ad google::LogMessage::Fail() at ??:0
@ 0x7f9c75fbe88a google::LogMessage::SendToLog() at ??:0
@ 0x7f9c75fbb4de google::LogMessage::Flush() at ??:0
@ 0x7f9c75fbca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9c7dee00f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9c7dee98b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9c77e890d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9c77e7e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9c78ce26db start_thread at ??:0
@ 0x7f9c74ab961f clone at ??:0
W20260620 19:10:27.027509 15642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:27.027710 15642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:27.027736 15642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:27.028964 15642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:27.029012 15642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:27.029039 15642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:10:27.030582 15642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:35085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=35737
--tserver_master_addrs=127.0.203.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:27.030851 15642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:27.031069 15642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:27.037371 15677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:27.037448 15679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:27.037381 15676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:27.037679 15642 server_base.cc:1034] running on GCE node
I20260620 19:10:27.037992 15642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:27.038304 15642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:27.039487 15642 hybrid_clock.cc:648] HybridClock initialized: now 1781982627039467 us; error 33 us; skew 500 ppm
Jun 20 19:10:27 dist-test-slave-wd30 krb5kdc[14497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:27.042459 15642 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:10:27.042945 15642 webserver.cc:460] Webserver started at http://127.0.203.1:35737/ using document root <none> and password file <none>
I20260620 19:10:27.043246 15642 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:27.043314 15642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:27.044749 15642 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:10:27.045523 15686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:27.045728 15642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:10:27.045828 15642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "4ac0bbcccaec4bc0bcb934cd315bc756"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-wd30"
server_key: "ddcf783531984b4816fb42342d335223"
server_key_iv: "abf841ec75990b4d4a67e554b61b9cad"
server_key_version: "encryptionkey@0"
I20260620 19:10:27.046232 15642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:27.067225 15642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:27.069813 15642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:27.069977 15642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:27.070240 15642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:27.070755 15693 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:10:27.072239 15642 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:10:27.072299 15642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 19:10:27.072331 15642 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:10:27.073153 15642 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:10:27.073197 15642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 19:10:27.073261 15693 tablet_bootstrap.cc:492] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756: Bootstrap starting.
I20260620 19:10:27.080165 15642 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:35085
I20260620 19:10:27.080315 15800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:35085 every 8 connection(s)
I20260620 19:10:27.080571 15642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:10:27.083417 15693 log.cc:824] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:27.085409 15693 tablet_bootstrap.cc:492] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:10:27.085683 15693 tablet_bootstrap.cc:492] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756: Bootstrap complete.
I20260620 19:10:27.086588 15693 ts_tablet_manager.cc:1397] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260620 19:10:27.087617 15693 raft_consensus.cc:348] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } } peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } }
I20260620 19:10:27.087888 15693 raft_consensus.cc:729] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ac0bbcccaec4bc0bcb934cd315bc756, State: Initialized, Role: FOLLOWER
I20260620 19:10:27.088059 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 15642
I20260620 19:10:27.088034 15693 consensus_queue.cc:260] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } } peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } }
I20260620 19:10:27.088315 15693 ts_tablet_manager.cc:1428] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260620 19:10:27.088356 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:37147
--local_ip_for_outbound_sockets=127.0.203.2
--tserver_master_addrs=127.0.203.62:42193
--webserver_port=33551
--webserver_interface=127.0.203.2
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:27.177111 15803 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:27.081237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.1:35105)
0620 19:10:27.081472 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:27.081528 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:27.082127 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:27.176944 (+ 94817us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":125,"threads_started":1}
W20260620 19:10:27.177389 15801 heartbeater.cc:644] Failed to heartbeat to 127.0.203.62:42193 (0 consecutive failures): Network error: Failed to ping master at 127.0.203.62:42193: Client connection negotiation failed: client connection to 127.0.203.62:42193: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:10:27.177764 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:27.177644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.1:46447)
0620 19:10:27.177708 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:27.177746 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 19:10:27.178105 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:27.178014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.1:44687)
0620 19:10:27.178060 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:27.178083 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 19:10:27.178207 15801 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:10:27.198395 15806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:27.198627 15806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:27.198679 15806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:27.199939 15806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:27.200002 15806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:27.200052 15806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:10:27.201558 15806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:37147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=33551
--tserver_master_addrs=127.0.203.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:27.201825 15806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:27.202095 15806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:27.207389 15814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:27.207378 15806 server_base.cc:1034] running on GCE node
W20260620 19:10:27.207414 15811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:27.207378 15812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:27.207870 15806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:27.208143 15806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:27.209296 15806 hybrid_clock.cc:648] HybridClock initialized: now 1781982627209293 us; error 33 us; skew 500 ppm
Jun 20 19:10:27 dist-test-slave-wd30 krb5kdc[14497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:27.211846 15806 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:10:27.212284 15806 webserver.cc:460] Webserver started at http://127.0.203.2:33551/ using document root <none> and password file <none>
I20260620 19:10:27.212524 15806 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:27.212591 15806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:27.213968 15806 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:10:27.214576 15821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:27.214769 15806 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:10:27.214870 15806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "5fc6abacdf114f00b6004264326ac657"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-wd30"
server_key: "68269f0ae8f81cd6017c3c62694896b0"
server_key_iv: "97ca673552e3f2e8b70879a222ef89c2"
server_key_version: "encryptionkey@0"
I20260620 19:10:27.215278 15806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:27.223650 15806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:27.225567 15806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:27.225725 15806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:27.225957 15806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:27.226258 15806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:27.226307 15806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:27.226349 15806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:27.226383 15806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:27.232478 15806 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:37147
I20260620 19:10:27.232515 15934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:37147 every 8 connection(s)
I20260620 19:10:27.232884 15806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:10:27.233701 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 15806
I20260620 19:10:27.234000 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:36067
--local_ip_for_outbound_sockets=127.0.203.3
--tserver_master_addrs=127.0.203.62:42193
--webserver_port=35733
--webserver_interface=127.0.203.3
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:10:27.234191 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:27.233583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.2:40435)
0620 19:10:27.233753 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:27.233837 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
W20260620 19:10:27.234407 15935 heartbeater.cc:644] Failed to heartbeat to 127.0.203.62:42193 (0 consecutive failures): Network error: Failed to ping master at 127.0.203.62:42193: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
I20260620 19:10:27.234686 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:27.234589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.2:47587)
0620 19:10:27.234638 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:27.234667 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 19:10:27.234953 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:27.234864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.2:50629)
0620 19:10:27.234910 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:27.234931 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 19:10:27.235052 15935 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:10:27.320200 15938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:27.320430 15938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:27.320458 15938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:27.321867 15938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:27.321938 15938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:27.321992 15938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:10:27.323918 15938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:36067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=35733
--tserver_master_addrs=127.0.203.62:42193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:27.324144 15938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:27.324342 15938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:27.329707 15946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:27.329702 15943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:27.329702 15944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:27.329900 15938 server_base.cc:1034] running on GCE node
I20260620 19:10:27.330153 15938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:27.330375 15938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:27.331568 15938 hybrid_clock.cc:648] HybridClock initialized: now 1781982627331555 us; error 29 us; skew 500 ppm
Jun 20 19:10:27 dist-test-slave-wd30 krb5kdc[14497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:27.334122 15938 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:10:27.334591 15938 webserver.cc:460] Webserver started at http://127.0.203.3:35733/ using document root <none> and password file <none>
I20260620 19:10:27.334847 15938 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:27.334918 15938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:27.336589 15938 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:27.337373 15953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:27.337687 15938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:27.337816 15938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "40c36c517a94406a82f67549435981fb"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-wd30"
server_key: "73f1133ca9543a3ae7fb9b5c6f9f9b61"
server_key_iv: "0ea6d2a128b81f5ec70ef1ff55335b8e"
server_key_version: "encryptionkey@0"
I20260620 19:10:27.338197 15938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:27.354715 15938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:27.356966 15938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:27.357133 15938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:27.357391 15938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:27.357718 15938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:27.357770 15938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:27.357815 15938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:27.357844 15938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:27.363786 15938 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:36067
I20260620 19:10:27.363895 16066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:36067 every 8 connection(s)
I20260620 19:10:27.364189 15938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:27.365204 16069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:27.364857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.3:33289)
0620 19:10:27.365004 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:27.365073 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
W20260620 19:10:27.365609 16067 heartbeater.cc:644] Failed to heartbeat to 127.0.203.62:42193 (0 consecutive failures): Network error: Failed to ping master at 127.0.203.62:42193: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
I20260620 19:10:27.365873 16069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:27.365790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.3:47225)
0620 19:10:27.365830 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:27.365856 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 19:10:27.366125 16069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:27.366058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.3:56977)
0620 19:10:27.366092 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:27.366112 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 19:10:27.366231 16067 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:10:27.369658 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 15938
I20260620 19:10:28.178779 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.178543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.1:33131)
0620 19:10:28.178683 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.178719 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260620 19:10:28.235721 16072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.235451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.2:59379)
0620 19:10:28.235618 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.235661 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":63,"threads_started":1}
I20260620 19:10:28.270607 16073 raft_consensus.cc:482] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:28.270725 16073 raft_consensus.cc:504] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } } peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } }
I20260620 19:10:28.271100 16073 leader_election.cc:288] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c36c517a94406a82f67549435981fb (127.0.203.3:36067), 5fc6abacdf114f00b6004264326ac657 (127.0.203.2:37147)
Jun 20 19:10:28 dist-test-slave-wd30 krb5kdc[14497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.2@KRBTEST.COM
Jun 20 19:10:28 dist-test-slave-wd30 krb5kdc[14497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.3@KRBTEST.COM
I20260620 19:10:28.281392 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.271296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:37147 (local address 127.0.203.1:46657)
0620 19:10:28.271449 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.271462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:28.271523 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:28.272310 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:28.272313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:28.272714 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:28.272898 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:28.272904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.274042 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.274043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:28.274783 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:28.274785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.275449 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.275450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:28.275523 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:28.276102 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:28.276109 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:28.277719 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:28.279780 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:28.279787 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:28.279789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:28.280063 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:28.280355 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:28.280356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:28.280356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:28.280517 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:28.280890 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:28.280896 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:28.281018 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 19:10:28.281119 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 19:10:28.281299 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":57}
I20260620 19:10:28.281405 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.271424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:46657 (local address 127.0.203.2:37147)
0620 19:10:28.271611 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:28.271627 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:28.271671 (+ 44us) server_negotiation.cc:407] Connection header received
0620 19:10:28.272092 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:28.272093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:28.272129 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:28.272287 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:28.273036 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.273935 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.274887 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.275340 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.275469 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:28.277824 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:28.277828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:28.277828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:28.277838 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:28.279678 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:28.280146 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:28.280148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:28.280149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:28.280285 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:28.280585 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:28.280586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:28.280586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:28.280815 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:28.280903 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:28.281167 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 19:10:28.281290 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":62,"threads_started":1}
I20260620 19:10:28.281462 16075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.271418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:57141 (local address 127.0.203.3:36067)
0620 19:10:28.271596 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:28.271607 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:28.271636 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:10:28.272068 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:28.272071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:28.272120 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:28.272269 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:28.273035 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.273913 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.274889 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.275312 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.275421 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:28.278220 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:28.278224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:28.278224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:28.278235 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:28.279925 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:28.280299 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:28.280302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:28.280303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:28.280417 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:28.280681 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:28.280682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:28.280682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:28.280900 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:28.280995 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:28.281197 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 19:10:28.281339 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
I20260620 19:10:28.281520 16074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.271296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:36067 (local address 127.0.203.1:57141)
0620 19:10:28.271459 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.271470 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:28.271534 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:28.272310 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:28.272312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:28.272714 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:28.272898 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:28.272904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.274010 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.274011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:28.274780 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:28.274782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.275411 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.275412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:28.275518 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:28.276145 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:28.276150 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:28.278132 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:28.280002 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:28.280003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:28.280003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:28.280214 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:28.280503 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:28.280516 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:28.280517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:28.280614 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:28.280979 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:28.280980 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:28.281025 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 19:10:28.281144 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 19:10:28.281377 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":154,"thread_start_us":77,"threads_started":1}
W20260620 19:10:28.282756 15690 leader_election.cc:341] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40c36c517a94406a82f67549435981fb (127.0.203.3:36067): Not found: Tablet not found: 8351285b2f2e46c491d78c5bb45873a4
W20260620 19:10:28.282864 15689 leader_election.cc:341] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5fc6abacdf114f00b6004264326ac657 (127.0.203.2:37147): Not found: Tablet not found: 8351285b2f2e46c491d78c5bb45873a4
I20260620 19:10:28.282907 15689 leader_election.cc:302] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ac0bbcccaec4bc0bcb934cd315bc756; no voters: 40c36c517a94406a82f67549435981fb, 5fc6abacdf114f00b6004264326ac657
I20260620 19:10:28.283036 16073 raft_consensus.cc:2726] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:10:28.366703 16077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.366518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.3:48029)
0620 19:10:28.366630 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.366669 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260620 19:10:29.179579 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:29.179342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.1:46529)
0620 19:10:29.179491 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:29.179535 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
I20260620 19:10:29.236858 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:29.236560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.2:55049)
0620 19:10:29.236765 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:29.236810 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
I20260620 19:10:29.367465 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:29.367276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.3:46571)
0620 19:10:29.367394 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:29.367429 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260620 19:10:30.180248 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.180023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.1:53525)
0620 19:10:30.180160 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:30.180197 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
I20260620 19:10:30.237905 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.237636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.2:38615)
0620 19:10:30.237803 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:30.237851 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":69,"threads_started":1}
I20260620 19:10:30.336122 16083 raft_consensus.cc:482] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:30.336257 16083 raft_consensus.cc:504] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } } peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } }
I20260620 19:10:30.336444 16083 leader_election.cc:288] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c36c517a94406a82f67549435981fb (127.0.203.3:36067), 5fc6abacdf114f00b6004264326ac657 (127.0.203.2:37147)
W20260620 19:10:30.336887 15690 leader_election.cc:341] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40c36c517a94406a82f67549435981fb (127.0.203.3:36067): Not found: Tablet not found: 8351285b2f2e46c491d78c5bb45873a4
W20260620 19:10:30.336982 15689 leader_election.cc:341] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5fc6abacdf114f00b6004264326ac657 (127.0.203.2:37147): Not found: Tablet not found: 8351285b2f2e46c491d78c5bb45873a4
I20260620 19:10:30.337045 15689 leader_election.cc:302] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ac0bbcccaec4bc0bcb934cd315bc756; no voters: 40c36c517a94406a82f67549435981fb, 5fc6abacdf114f00b6004264326ac657
I20260620 19:10:30.337131 16083 raft_consensus.cc:2726] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:10:30.368161 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.367945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.3:33717)
0620 19:10:30.368082 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:30.368125 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260620 19:10:31.180966 16085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:31.180734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.1:35073)
0620 19:10:31.180872 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:31.180922 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260620 19:10:31.238677 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:31.238432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.2:40123)
0620 19:10:31.238582 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:31.238629 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
I20260620 19:10:31.368746 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:31.368572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.3:45143)
0620 19:10:31.368681 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:31.368712 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
I20260620 19:10:32.181661 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:32.181456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.1:54379)
0620 19:10:32.181581 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:32.181617 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260620 19:10:32.239763 16089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:32.239388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.2:44465)
0620 19:10:32.239614 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:32.239690 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
I20260620 19:10:32.369405 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:32.369192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.3:47163)
0620 19:10:32.369326 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:32.369363 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260620 19:10:33.182484 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:33.182250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.1:54267)
0620 19:10:33.182399 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:33.182438 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260620 19:10:33.240873 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:33.240533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.2:39505)
0620 19:10:33.240723 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:33.240802 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":78,"threads_started":1}
I20260620 19:10:33.370131 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:33.369937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.3:46785)
0620 19:10:33.370063 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:33.370093 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260620 19:10:33.417410 16094 raft_consensus.cc:482] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:33.417531 16094 raft_consensus.cc:504] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } } peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } }
I20260620 19:10:33.417708 16094 leader_election.cc:288] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c36c517a94406a82f67549435981fb (127.0.203.3:36067), 5fc6abacdf114f00b6004264326ac657 (127.0.203.2:37147)
W20260620 19:10:33.418176 15689 leader_election.cc:341] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5fc6abacdf114f00b6004264326ac657 (127.0.203.2:37147): Not found: Tablet not found: 8351285b2f2e46c491d78c5bb45873a4
W20260620 19:10:33.418246 15690 leader_election.cc:341] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40c36c517a94406a82f67549435981fb (127.0.203.3:36067): Not found: Tablet not found: 8351285b2f2e46c491d78c5bb45873a4
I20260620 19:10:33.418339 15690 leader_election.cc:302] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ac0bbcccaec4bc0bcb934cd315bc756; no voters: 40c36c517a94406a82f67549435981fb, 5fc6abacdf114f00b6004264326ac657
I20260620 19:10:33.418431 16094 raft_consensus.cc:2726] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:10:34.183209 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:34.182908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.1:57483)
0620 19:10:34.183065 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:34.183101 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
I20260620 19:10:34.241853 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:34.241557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.2:33453)
0620 19:10:34.241719 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:34.241782 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
I20260620 19:10:34.370764 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:34.370534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.3:52999)
0620 19:10:34.370689 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:34.370727 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
I20260620 19:10:35.183813 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:35.183599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.1:60095)
0620 19:10:35.183734 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:35.183765 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
I20260620 19:10:35.242915 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:35.242649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.2:43545)
0620 19:10:35.242816 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:35.242859 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260620 19:10:35.371531 16100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:35.371338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.3:53853)
0620 19:10:35.371459 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:35.371495 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260620 19:10:35.758028 16101 raft_consensus.cc:482] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:35.758174 16101 raft_consensus.cc:504] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } } peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } }
I20260620 19:10:35.758386 16101 leader_election.cc:288] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c36c517a94406a82f67549435981fb (127.0.203.3:36067), 5fc6abacdf114f00b6004264326ac657 (127.0.203.2:37147)
W20260620 19:10:35.758785 15689 leader_election.cc:341] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5fc6abacdf114f00b6004264326ac657 (127.0.203.2:37147): Not found: Tablet not found: 8351285b2f2e46c491d78c5bb45873a4
W20260620 19:10:35.758867 15690 leader_election.cc:341] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40c36c517a94406a82f67549435981fb (127.0.203.3:36067): Not found: Tablet not found: 8351285b2f2e46c491d78c5bb45873a4
I20260620 19:10:35.758929 15690 leader_election.cc:302] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ac0bbcccaec4bc0bcb934cd315bc756; no voters: 40c36c517a94406a82f67549435981fb, 5fc6abacdf114f00b6004264326ac657
I20260620 19:10:35.759032 16101 raft_consensus.cc:2726] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:10:36.184422 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:36.184212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.1:37695)
0620 19:10:36.184341 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:36.184376 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260620 19:10:36.243860 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:36.243599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.2:60537)
0620 19:10:36.243766 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:36.243808 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
I20260620 19:10:36.372123 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:36.371939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.3:34533)
0620 19:10:36.372051 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:36.372083 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260620 19:10:37.185178 16105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:37.184933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.1:48753)
0620 19:10:37.185088 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:37.185132 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
I20260620 19:10:37.244753 16106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:37.244539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.2:54275)
0620 19:10:37.244681 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:37.244708 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260620 19:10:37.372855 16107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:37.372644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.3:59413)
0620 19:10:37.372772 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:37.372807 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260620 19:10:38.038329 16108 raft_consensus.cc:482] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:38.038455 16108 raft_consensus.cc:504] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c36c517a94406a82f67549435981fb" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36067 } } peers { permanent_uuid: "5fc6abacdf114f00b6004264326ac657" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37147 } } peers { permanent_uuid: "4ac0bbcccaec4bc0bcb934cd315bc756" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35085 } }
I20260620 19:10:38.038682 16108 leader_election.cc:288] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c36c517a94406a82f67549435981fb (127.0.203.3:36067), 5fc6abacdf114f00b6004264326ac657 (127.0.203.2:37147)
W20260620 19:10:38.039069 15689 leader_election.cc:341] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5fc6abacdf114f00b6004264326ac657 (127.0.203.2:37147): Not found: Tablet not found: 8351285b2f2e46c491d78c5bb45873a4
W20260620 19:10:38.039178 15690 leader_election.cc:341] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40c36c517a94406a82f67549435981fb (127.0.203.3:36067): Not found: Tablet not found: 8351285b2f2e46c491d78c5bb45873a4
I20260620 19:10:38.039227 15690 leader_election.cc:302] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ac0bbcccaec4bc0bcb934cd315bc756; no voters: 40c36c517a94406a82f67549435981fb, 5fc6abacdf114f00b6004264326ac657
I20260620 19:10:38.039300 16108 raft_consensus.cc:2726] T 8351285b2f2e46c491d78c5bb45873a4 P 4ac0bbcccaec4bc0bcb934cd315bc756 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:10:38.185977 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:38.185688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.1:48967)
0620 19:10:38.185855 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:38.185911 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":65,"threads_started":1}
I20260620 19:10:38.245500 16110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:38.245263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.2:49277)
0620 19:10:38.245408 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:38.245452 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260620 19:10:38.373729 16111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:38.373436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.3:53475)
0620 19:10:38.373623 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:38.373669 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":88,"threads_started":1}
I20260620 19:10:39.186782 16112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:39.186508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.1:55669)
0620 19:10:39.186681 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:39.186735 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":81,"threads_started":1}
I20260620 19:10:39.246215 16113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:39.245993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.2:58425)
0620 19:10:39.246131 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:39.246167 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260620 19:10:39.374440 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:39.374227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.3:37527)
0620 19:10:39.374351 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:39.374393 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260620 19:10:40.187673 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:40.187428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.1:42631)
0620 19:10:40.187576 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:40.187618 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260620 19:10:40.246974 16116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:40.246752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.2:50175)
0620 19:10:40.246880 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:40.246926 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
I20260620 19:10:40.375214 16117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:40.374939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.3:55663)
0620 19:10:40.375068 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:40.375105 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260620 19:10:41.188333 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:41.188132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.1:60623)
0620 19:10:41.188254 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:41.188287 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260620 19:10:41.247807 16119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:41.247577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.2:55589)
0620 19:10:41.247715 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:41.247758 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260620 19:10:41.375926 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:41.375697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.3:49029)
0620 19:10:41.375838 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:41.375880 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260620 19:10:42.189162 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:42.188931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.1:39543)
0620 19:10:42.189081 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:42.189116 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":71,"threads_started":1}
I20260620 19:10:42.248546 16122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:42.248340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.2:57027)
0620 19:10:42.248460 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:42.248499 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260620 19:10:42.376727 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:42.376498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42193 (local address 127.0.203.3:33085)
0620 19:10:42.376642 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:42.376683 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
I20260620 19:10:42.378006 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 15642
I20260620 19:10:42.382623 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 15806
I20260620 19:10:42.386937 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 15938
2026-06-20T19:10:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18632 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:10:42 dist-test-slave-wd30 krb5kdc[16126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:10:42 dist-test-slave-wd30 krb5kdc[16126](info): set up 2 sockets
Jun 20 19:10:42 dist-test-slave-wd30 krb5kdc[16126](info): commencing operation
krb5kdc: starting...
W20260620 19:10:43.439467 812 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:10:43 dist-test-slave-wd30 krb5kdc[16126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:10:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:10:43.557399 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:46105
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:46105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:43.637362 16139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:43.637677 16139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:43.637705 16139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:43.638839 16139 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 19:10:43.638875 16139 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 19:10:43.638887 16139 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 19:10:43.638899 16139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:43.638911 16139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:43.638924 16139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:43.638937 16139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:43.638954 16139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:43.640369 16139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:46105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:46105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:43.640568 16139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:43.640741 16139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:43.645777 16145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:43.645808 16139 server_base.cc:1034] running on GCE node
W20260620 19:10:43.645789 16147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:43.645865 16144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:43.646247 16139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:43.646569 16139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:43.647742 16139 hybrid_clock.cc:648] HybridClock initialized: now 1781982643647728 us; error 29 us; skew 500 ppm
Jun 20 19:10:43 dist-test-slave-wd30 krb5kdc[16126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:43.650259 16139 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:10:43.650761 16139 webserver.cc:460] Webserver started at http://127.0.203.62:34311/ using document root <none> and password file <none>
I20260620 19:10:43.651005 16139 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:43.651073 16139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:43.651255 16139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:43.652383 16139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "ece60f07c20b4e4bbbce803100626d68"
format_stamp: "Formatted at 2026-06-20 19:10:43 on dist-test-slave-wd30"
server_key: "b3d36dbbc1da725dd22d46696fad1444"
server_key_iv: "ccd965a1b17a61a273f0d72daa2912e5"
server_key_version: "encryptionkey@0"
I20260620 19:10:43.652755 16139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "ece60f07c20b4e4bbbce803100626d68"
format_stamp: "Formatted at 2026-06-20 19:10:43 on dist-test-slave-wd30"
server_key: "b3d36dbbc1da725dd22d46696fad1444"
server_key_iv: "ccd965a1b17a61a273f0d72daa2912e5"
server_key_version: "encryptionkey@0"
I20260620 19:10:43.654109 16139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:10:43.654826 16154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:43.654999 16139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:10:43.655102 16139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "ece60f07c20b4e4bbbce803100626d68"
format_stamp: "Formatted at 2026-06-20 19:10:43 on dist-test-slave-wd30"
server_key: "b3d36dbbc1da725dd22d46696fad1444"
server_key_iv: "ccd965a1b17a61a273f0d72daa2912e5"
server_key_version: "encryptionkey@0"
I20260620 19:10:43.655233 16139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:43.668463 16139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:43.670334 16139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:43.670506 16139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:43.674768 16139 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:46105
I20260620 19:10:43.674808 16206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:46105 every 8 connection(s)
I20260620 19:10:43.675209 16139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:43.676033 16207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:43.678237 16207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: Bootstrap starting.
I20260620 19:10:43.678778 16207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:43.679008 16207 log.cc:824] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:43.679664 16207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: No bootstrap required, opened a new log
I20260620 19:10:43.681125 16207 raft_consensus.cc:348] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece60f07c20b4e4bbbce803100626d68" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 46105 } }
I20260620 19:10:43.681267 16207 raft_consensus.cc:374] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:43.681290 16207 raft_consensus.cc:729] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece60f07c20b4e4bbbce803100626d68, State: Initialized, Role: FOLLOWER
I20260620 19:10:43.681387 16207 consensus_queue.cc:260] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece60f07c20b4e4bbbce803100626d68" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 46105 } }
I20260620 19:10:43.681438 16207 raft_consensus.cc:388] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:43.681458 16207 raft_consensus.cc:482] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:43.681484 16207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:43.681931 16207 raft_consensus.cc:504] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece60f07c20b4e4bbbce803100626d68" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 46105 } }
I20260620 19:10:43.682011 16207 leader_election.cc:302] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ece60f07c20b4e4bbbce803100626d68; no voters:
I20260620 19:10:43.682145 16207 leader_election.cc:288] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:43.682193 16210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:43.682323 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 16139
I20260620 19:10:43.682343 16210 raft_consensus.cc:686] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 1 LEADER]: Becoming Leader. State: Replica: ece60f07c20b4e4bbbce803100626d68, State: Running, Role: LEADER
I20260620 19:10:43.682402 16207 sys_catalog.cc:564] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:43.682425 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:10:43.682688 812 external_mini_cluster.cc:1428] Setting key 99f94791ebf05877f8076c4345873e6e
I20260620 19:10:43.682432 16210 consensus_queue.cc:237] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece60f07c20b4e4bbbce803100626d68" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 46105 } }
I20260620 19:10:43.683068 16211 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ece60f07c20b4e4bbbce803100626d68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece60f07c20b4e4bbbce803100626d68" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 46105 } } }
I20260620 19:10:43.683171 16211 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:43.683332 16212 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ece60f07c20b4e4bbbce803100626d68. Latest consensus state: current_term: 1 leader_uuid: "ece60f07c20b4e4bbbce803100626d68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece60f07c20b4e4bbbce803100626d68" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 46105 } } }
I20260620 19:10:43.683393 16212 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:43.683374 16220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:43.684278 16220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:43.686600 16220 catalog_manager.cc:1347] Generated new cluster ID: 7e1dbb4029a64b3385c7d43ee0611b2e
I20260620 19:10:43.686671 16220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:10:43 dist-test-slave-wd30 krb5kdc[16126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:43.694203 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:43.683199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47916 (local address 127.0.203.62:46105)
0620 19:10:43.683456 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:43.683466 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:43.683500 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:10:43.683617 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:43.683625 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:43.683678 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:43.684642 (+ 964us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:43.684767 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:43.686081 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:43.687006 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:43.687684 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:43.687792 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:43.690539 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:43.690543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:43.690543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:43.690556 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:43.692530 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:43.692944 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:43.692947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:43.692947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:43.693052 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:43.693272 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:43.693272 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:43.693273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:43.693560 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:43.693701 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:43.693832 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 19:10:43.693960 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":71,"threads_started":1}
I20260620 19:10:43.705683 16220 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:43.706228 16220 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:43.708837 16220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: Generated new TSK 0
I20260620 19:10:43.709024 16220 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
I20260620 19:10:43.735790 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:46105
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:43.817749 16233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:43.817981 16233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:43.818015 16233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:43.819185 16233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:43.819252 16233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:43.819298 16233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:10:43.820756 16233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:46105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:43.821040 16233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:43.821272 16233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:43.826169 16241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:43.826180 16239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:43.826269 16238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:43.826292 16233 server_base.cc:1034] running on GCE node
I20260620 19:10:43.826623 16233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:43.826885 16233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:43.828047 16233 hybrid_clock.cc:648] HybridClock initialized: now 1781982643828025 us; error 35 us; skew 500 ppm
Jun 20 19:10:43 dist-test-slave-wd30 krb5kdc[16126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:43.830488 16233 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:10:43.830854 16233 webserver.cc:460] Webserver started at http://127.0.203.1:38671/ using document root <none> and password file <none>
I20260620 19:10:43.831048 16233 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:43.831089 16233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:43.831243 16233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:43.832222 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "1e931308008d429cab5bc8a64877424f"
format_stamp: "Formatted at 2026-06-20 19:10:43 on dist-test-slave-wd30"
server_key: "00dfdc59af68ef5c9eb22746ffd4cccd"
server_key_iv: "ba4ba3634e53a0a0dbd1cb72d1f0175c"
server_key_version: "encryptionkey@0"
I20260620 19:10:43.832504 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "1e931308008d429cab5bc8a64877424f"
format_stamp: "Formatted at 2026-06-20 19:10:43 on dist-test-slave-wd30"
server_key: "00dfdc59af68ef5c9eb22746ffd4cccd"
server_key_iv: "ba4ba3634e53a0a0dbd1cb72d1f0175c"
server_key_version: "encryptionkey@0"
I20260620 19:10:43.833705 16233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:10:43.834368 16248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:43.834651 16233 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:43.834753 16233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "1e931308008d429cab5bc8a64877424f"
format_stamp: "Formatted at 2026-06-20 19:10:43 on dist-test-slave-wd30"
server_key: "00dfdc59af68ef5c9eb22746ffd4cccd"
server_key_iv: "ba4ba3634e53a0a0dbd1cb72d1f0175c"
server_key_version: "encryptionkey@0"
I20260620 19:10:43.834825 16233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:43.854051 16233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:43.856432 16233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:43.856611 16233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:43.856859 16233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:43.857189 16233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:43.857244 16233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:43.857283 16233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:43.857311 16233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:43.863209 16233 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:39807
I20260620 19:10:43.863267 16361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:39807 every 8 connection(s)
I20260620 19:10:43.863613 16233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:10:43 dist-test-slave-wd30 krb5kdc[16126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:43.870889 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 16233
I20260620 19:10:43.871011 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:10:43.871276 812 external_mini_cluster.cc:1428] Setting key 2af5f6738542c576b4980d6cd5fee6e7
I20260620 19:10:43.873237 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:43.864348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:39393 (local address 127.0.203.62:46105)
0620 19:10:43.864397 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:43.864399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:43.864587 (+ 188us) server_negotiation.cc:407] Connection header received
0620 19:10:43.865112 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:43.865114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:43.865125 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:43.865182 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:43.866124 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:43.866628 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:43.867714 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:43.867916 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:43.868031 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:43.870035 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:43.870040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:43.870040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:43.870049 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:43.871712 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:43.872241 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:43.872242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:43.872242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:43.872295 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:43.872611 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:43.872611 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:43.872612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:43.872714 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:43.872772 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:43.873110 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 19:10:43.873174 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:10:43.873580 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:43.864310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.1:39393)
0620 19:10:43.864446 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:43.864498 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:43.865007 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:43.865252 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:43.865256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:43.865832 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:43.866022 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:43.866026 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:43.866723 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:43.866725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:43.867619 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:43.867622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:43.868042 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:43.868043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:43.868168 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:43.868598 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:43.868606 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:43.869919 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:43.871783 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:43.871790 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:43.871792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:43.872159 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:43.872393 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:43.872393 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:43.872394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:43.872540 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:43.872785 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:43.872792 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:43.872965 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 19:10:43.873092 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 19:10:43.873279 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
I20260620 19:10:43.874190 16362 heartbeater.cc:347] Connected to a master server at 127.0.203.62:46105
I20260620 19:10:43.874292 16362 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:43.874509 16362 heartbeater.cc:511] Master 127.0.203.62:46105 requested a full tablet report, sending...
I20260620 19:10:43.875090 16171 ts_manager.cc:194] Registered new tserver with Master: 1e931308008d429cab5bc8a64877424f (127.0.203.1:39807)
I20260620 19:10:43.875810 16171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:39393
WARNING: no policy specified for kudu/127.0.203.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
I20260620 19:10:43.898358 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:46105
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:43.979297 16367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:43.979511 16367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:43.979535 16367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:43.980664 16367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:43.980698 16367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:43.980726 16367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:10:43.982103 16367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:46105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:43.982287 16367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:43.982573 16367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:43.987681 16372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:43.987710 16373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:43.987715 16375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:43.987938 16367 server_base.cc:1034] running on GCE node
I20260620 19:10:43.988131 16367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:43.988359 16367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:43.989531 16367 hybrid_clock.cc:648] HybridClock initialized: now 1781982643989512 us; error 34 us; skew 500 ppm
Jun 20 19:10:43 dist-test-slave-wd30 krb5kdc[16126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:43.992259 16367 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:10:43.992758 16367 webserver.cc:460] Webserver started at http://127.0.203.2:40031/ using document root <none> and password file <none>
I20260620 19:10:43.992983 16367 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:43.993050 16367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:43.993199 16367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:43.994256 16367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0"
format_stamp: "Formatted at 2026-06-20 19:10:43 on dist-test-slave-wd30"
server_key: "62684ef19fc6df101406cbde57e82130"
server_key_iv: "779c3f08b3e82327c91ffdf55e4f0a85"
server_key_version: "encryptionkey@0"
I20260620 19:10:43.994623 16367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0"
format_stamp: "Formatted at 2026-06-20 19:10:43 on dist-test-slave-wd30"
server_key: "62684ef19fc6df101406cbde57e82130"
server_key_iv: "779c3f08b3e82327c91ffdf55e4f0a85"
server_key_version: "encryptionkey@0"
I20260620 19:10:43.996057 16367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:10:43.996855 16382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:43.997051 16367 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:43.997154 16367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0"
format_stamp: "Formatted at 2026-06-20 19:10:43 on dist-test-slave-wd30"
server_key: "62684ef19fc6df101406cbde57e82130"
server_key_iv: "779c3f08b3e82327c91ffdf55e4f0a85"
server_key_version: "encryptionkey@0"
I20260620 19:10:43.997226 16367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:44.013954 16367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:44.016379 16367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:44.016563 16367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:44.016816 16367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:44.017129 16367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:44.017184 16367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:44.017225 16367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:44.017273 16367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:44.023322 16367 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:44409
I20260620 19:10:44.023365 16495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:44409 every 8 connection(s)
I20260620 19:10:44.023687 16367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:10:44 dist-test-slave-wd30 krb5kdc[16126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:44.032475 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:44.024463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:38353 (local address 127.0.203.62:46105)
0620 19:10:44.024529 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:44.024531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:44.024691 (+ 160us) server_negotiation.cc:407] Connection header received
0620 19:10:44.025146 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:44.025147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:44.025156 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:44.025223 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:44.026135 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.026631 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.027647 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.027834 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.027890 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.029971 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:44.029973 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:44.029973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:44.029980 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:44.031252 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:44.031636 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:44.031638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:44.031638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:44.031726 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:44.031956 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:44.031957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:44.031957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:44.032061 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:44.032117 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:44.032332 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 19:10:44.032409 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:10:44.032665 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:44.024380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.2:38353)
0620 19:10:44.024558 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:44.024601 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:44.025065 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:44.025276 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:44.025279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:44.025850 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:44.026047 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:44.026052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.026710 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.026711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:44.027573 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:44.027576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.027905 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.027906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:44.028004 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.028556 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:44.028565 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:44.029889 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:44.031317 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:44.031323 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:44.031325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:44.031539 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:44.031791 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:44.031791 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:44.031791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:44.031879 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:44.032117 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:44.032121 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:44.032227 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 19:10:44.032304 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 19:10:44.032416 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
I20260620 19:10:44.033190 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 16367
I20260620 19:10:44.033198 16496 heartbeater.cc:347] Connected to a master server at 127.0.203.62:46105
I20260620 19:10:44.033298 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:10:44.033303 16496 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:44.033471 16496 heartbeater.cc:511] Master 127.0.203.62:46105 requested a full tablet report, sending...
I20260620 19:10:44.033519 812 external_mini_cluster.cc:1428] Setting key 484264dbb5ecf53a3e2ce1f47dc20b1a
I20260620 19:10:44.033874 16171 ts_manager.cc:194] Registered new tserver with Master: f10fd16e9d0e4285b4c8de19cb7af0e0 (127.0.203.2:44409)
I20260620 19:10:44.034407 16171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:38353
WARNING: no policy specified for kudu/127.0.203.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
I20260620 19:10:44.058117 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:46105
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:44.138741 16501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:44.138971 16501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:44.139009 16501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:44.140192 16501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:44.140265 16501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:44.140311 16501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:10:44.141755 16501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:46105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:10:44.142033 16501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:44.142266 16501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:44.147217 16509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:44.147218 16507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:44.147254 16506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:44.147341 16501 server_base.cc:1034] running on GCE node
I20260620 19:10:44.147665 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:44.147917 16501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:44.149080 16501 hybrid_clock.cc:648] HybridClock initialized: now 1781982644149064 us; error 28 us; skew 500 ppm
Jun 20 19:10:44 dist-test-slave-wd30 krb5kdc[16126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:44.151588 16501 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:10:44.152030 16501 webserver.cc:460] Webserver started at http://127.0.203.3:43667/ using document root <none> and password file <none>
I20260620 19:10:44.152480 16501 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:44.152544 16501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:44.152652 16501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:44.153661 16501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "d6785bfe80704762a95cc9bf281dd210"
format_stamp: "Formatted at 2026-06-20 19:10:44 on dist-test-slave-wd30"
server_key: "136c180a87920cc264ae1ea137fd76bd"
server_key_iv: "7971f97415af26e7935a35e03698bb74"
server_key_version: "encryptionkey@0"
I20260620 19:10:44.153966 16501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "d6785bfe80704762a95cc9bf281dd210"
format_stamp: "Formatted at 2026-06-20 19:10:44 on dist-test-slave-wd30"
server_key: "136c180a87920cc264ae1ea137fd76bd"
server_key_iv: "7971f97415af26e7935a35e03698bb74"
server_key_version: "encryptionkey@0"
I20260620 19:10:44.155267 16501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:10:44.156044 16516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:44.156208 16501 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:44.156296 16501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "d6785bfe80704762a95cc9bf281dd210"
format_stamp: "Formatted at 2026-06-20 19:10:44 on dist-test-slave-wd30"
server_key: "136c180a87920cc264ae1ea137fd76bd"
server_key_iv: "7971f97415af26e7935a35e03698bb74"
server_key_version: "encryptionkey@0"
I20260620 19:10:44.156355 16501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:44.168423 16501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:44.170672 16501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:44.170799 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:44.170970 16501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:44.171332 16501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:44.171370 16501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:44.171394 16501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:44.171437 16501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:44.177189 16501 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:43299
I20260620 19:10:44.177274 16629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:43299 every 8 connection(s)
I20260620 19:10:44.177572 16501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:44.183022 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 16501
I20260620 19:10:44.183198 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:10:44.183445 812 external_mini_cluster.cc:1428] Setting key 39463220adb826e84e84348b1dd75c97
Jun 20 19:10:44 dist-test-slave-wd30 krb5kdc[16126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:10:44.186731 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:44.178341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:39485 (local address 127.0.203.62:46105)
0620 19:10:44.178408 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:44.178410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:44.178569 (+ 159us) server_negotiation.cc:407] Connection header received
0620 19:10:44.179044 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:44.179045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:44.179054 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:44.179153 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:44.180100 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.180585 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.181578 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.181786 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.181857 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.184215 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:44.184216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:44.184217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:44.184225 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:44.185528 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:44.185932 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:44.185934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:44.185934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:44.185991 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:44.186211 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:44.186212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:44.186212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:44.186321 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:44.186373 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:44.186601 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 19:10:44.186668 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:10:44.186918 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:44.178291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.3:39485)
0620 19:10:44.178445 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:44.178486 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:44.178965 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:44.179224 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:44.179228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:44.179830 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:44.180023 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:44.180028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.180668 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.180669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:44.181475 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:44.181477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.181871 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.181871 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:44.181991 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.182483 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:44.182491 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:44.184120 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:44.185600 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:44.185606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:44.185607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:44.185820 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:44.186059 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:44.186059 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:44.186060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:44.186155 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:44.186376 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:44.186380 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:44.186493 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 19:10:44.186573 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 19:10:44.186687 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260620 19:10:44.187433 16630 heartbeater.cc:347] Connected to a master server at 127.0.203.62:46105
I20260620 19:10:44.187529 16630 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:44.187708 16630 heartbeater.cc:511] Master 127.0.203.62:46105 requested a full tablet report, sending...
I20260620 19:10:44.188071 16171 ts_manager.cc:194] Registered new tserver with Master: d6785bfe80704762a95cc9bf281dd210 (127.0.203.3:43299)
I20260620 19:10:44.188403 16171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:39485
I20260620 19:10:44.195274 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:10:44.202078 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:44.196595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47918 (local address 127.0.203.62:46105)
0620 19:10:44.196664 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:44.196666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:44.196677 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:10:44.196732 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:44.196733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:44.196743 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:44.196788 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:44.197298 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.197772 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.198494 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.198677 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.198740 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.199940 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:44.199942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:44.199942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:44.199962 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:44.201193 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:44.201491 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:44.201492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:44.201492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:44.201540 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:44.201686 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:44.201686 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:44.201686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:44.201813 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:44.201882 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:44.201964 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 19:10:44.202018 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:10:44.203715 16171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:10:44.205180 16171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:44.211591 16643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:44.207638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53408 (local address 127.0.203.2:44409)
0620 19:10:44.207815 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:44.207820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:44.207830 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:10:44.207870 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:44.207872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:44.207888 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:44.208043 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:44.208119 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.209231 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.210406 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.211167 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.211244 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.211306 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:44.211445 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 19:10:44.211496 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":69,"threads_started":1}
I20260620 19:10:44.211589 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:44.207523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:44409 (local address 127.0.0.1:53408)
0620 19:10:44.207695 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:44.207706 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:44.207773 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:44.207991 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:44.207994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:44.207997 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:44.208067 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:44.208069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.209351 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.209352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:44.210323 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:44.210326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.211241 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.211242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:44.211327 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.211341 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 19:10:44.211415 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:10:44.211474 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":22,"thread_start_us":42,"threads_started":1}
I20260620 19:10:44.211596 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:44.207320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:39807 (local address 127.0.0.1:37178)
0620 19:10:44.207619 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:44.207663 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:44.207728 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:44.208367 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:44.208368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:44.208369 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:44.208407 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:44.208409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.209565 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.209566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:44.210601 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:44.210603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.211371 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.211372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:44.211433 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.211436 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:10:44.211481 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 19:10:44.211521 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":1,"thread_start_us":67,"threads_started":1}
I20260620 19:10:44.211769 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:44.207421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37178 (local address 127.0.203.1:39807)
0620 19:10:44.208230 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:44.208234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:44.208243 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:10:44.208276 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:44.208278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:44.208293 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:44.208369 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:44.208471 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.209471 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.210678 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.211203 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.211272 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.211338 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:44.211634 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 19:10:44.211684 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":83,"threads_started":1}
I20260620 19:10:44.211779 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:44.207526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51906 (local address 127.0.203.3:43299)
0620 19:10:44.207943 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:44.207947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:44.207956 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:10:44.207993 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:44.207995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:44.208010 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:44.208098 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:44.208319 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.209057 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.210506 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.211186 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.211253 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.211306 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:44.211632 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 19:10:44.211681 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":80,"threads_started":1}
I20260620 19:10:44.212006 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:44.207337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:43299 (local address 127.0.0.1:51906)
0620 19:10:44.207619 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:44.207663 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:44.207728 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:44.208183 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:44.208184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:44.208186 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:44.208242 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:44.208244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.209155 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.209156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:44.210433 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:44.210435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.211348 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.211349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:44.211419 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.211424 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:10:44.211478 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 19:10:44.211521 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":67,"threads_started":1}
I20260620 19:10:44.212713 16564 tablet_service.cc:1467] Processing CreateTablet for tablet da2f18c8a22440bf8b0b8f46b09cb1c2 (DEFAULT_TABLE table=test-table [id=00dfefb1923e4fbfbb2769ccea5b45a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:44.212713 16296 tablet_service.cc:1467] Processing CreateTablet for tablet da2f18c8a22440bf8b0b8f46b09cb1c2 (DEFAULT_TABLE table=test-table [id=00dfefb1923e4fbfbb2769ccea5b45a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:44.212783 16430 tablet_service.cc:1467] Processing CreateTablet for tablet da2f18c8a22440bf8b0b8f46b09cb1c2 (DEFAULT_TABLE table=test-table [id=00dfefb1923e4fbfbb2769ccea5b45a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:44.213065 16296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da2f18c8a22440bf8b0b8f46b09cb1c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:44.213065 16564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da2f18c8a22440bf8b0b8f46b09cb1c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:44.213088 16430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da2f18c8a22440bf8b0b8f46b09cb1c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:44.215751 16644 tablet_bootstrap.cc:492] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0: Bootstrap starting.
I20260620 19:10:44.216301 16645 tablet_bootstrap.cc:492] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f: Bootstrap starting.
I20260620 19:10:44.216403 16644 tablet_bootstrap.cc:654] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:44.216506 16646 tablet_bootstrap.cc:492] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210: Bootstrap starting.
I20260620 19:10:44.217020 16645 tablet_bootstrap.cc:654] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:44.217028 16644 log.cc:824] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:44.217061 16646 tablet_bootstrap.cc:654] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:44.217541 16645 log.cc:824] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:44.217728 16646 log.cc:824] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:44.217875 16644 tablet_bootstrap.cc:492] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0: No bootstrap required, opened a new log
I20260620 19:10:44.217978 16644 ts_tablet_manager.cc:1397] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:44.218241 16645 tablet_bootstrap.cc:492] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f: No bootstrap required, opened a new log
I20260620 19:10:44.218293 16645 ts_tablet_manager.cc:1397] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:44.219615 16644 raft_consensus.cc:348] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } } peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } }
I20260620 19:10:44.219784 16644 raft_consensus.cc:374] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:44.219838 16644 raft_consensus.cc:729] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f10fd16e9d0e4285b4c8de19cb7af0e0, State: Initialized, Role: FOLLOWER
I20260620 19:10:44.219960 16644 consensus_queue.cc:260] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } } peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } }
I20260620 19:10:44.219919 16645 raft_consensus.cc:348] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } } peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } }
I20260620 19:10:44.220062 16645 raft_consensus.cc:374] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:44.220124 16645 raft_consensus.cc:729] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e931308008d429cab5bc8a64877424f, State: Initialized, Role: FOLLOWER
I20260620 19:10:44.220247 16644 ts_tablet_manager.cc:1428] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:10:44.220273 16496 heartbeater.cc:503] Master 127.0.203.62:46105 was elected leader, sending a full tablet report...
I20260620 19:10:44.220288 16645 consensus_queue.cc:260] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } } peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } }
I20260620 19:10:44.220549 16645 ts_tablet_manager.cc:1428] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:44.220582 16362 heartbeater.cc:503] Master 127.0.203.62:46105 was elected leader, sending a full tablet report...
I20260620 19:10:44.220852 16646 tablet_bootstrap.cc:492] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210: No bootstrap required, opened a new log
I20260620 19:10:44.220927 16646 ts_tablet_manager.cc:1397] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 19:10:44.222412 16646 raft_consensus.cc:348] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } } peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } }
I20260620 19:10:44.222543 16646 raft_consensus.cc:374] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:44.222564 16646 raft_consensus.cc:729] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6785bfe80704762a95cc9bf281dd210, State: Initialized, Role: FOLLOWER
I20260620 19:10:44.222692 16646 consensus_queue.cc:260] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } } peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } }
I20260620 19:10:44.222992 16646 ts_tablet_manager.cc:1428] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:44.223012 16630 heartbeater.cc:503] Master 127.0.203.62:46105 was elected leader, sending a full tablet report...
W20260620 19:10:44.274288 16497 tablet.cc:2369] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:10:44.305392 16652 raft_consensus.cc:482] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:44.305519 16652 raft_consensus.cc:504] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } } peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } }
I20260620 19:10:44.305908 16652 leader_election.cc:288] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e931308008d429cab5bc8a64877424f (127.0.203.1:39807), f10fd16e9d0e4285b4c8de19cb7af0e0 (127.0.203.2:44409)
I20260620 19:10:44.310138 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:44.306201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:55529 (local address 127.0.203.1:39807)
0620 19:10:44.306251 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:44.306253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:44.306264 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:10:44.306367 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:44.306369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:44.306383 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:44.306446 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:44.306671 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.307458 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.309156 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.309731 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.309814 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.309826 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:44.310013 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 19:10:44.310068 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 19:10:44.310138 16643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:44.306297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:41763 (local address 127.0.203.2:44409)
0620 19:10:44.306351 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:44.306353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:44.306514 (+ 161us) server_negotiation.cc:407] Connection header received
0620 19:10:44.306550 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:44.306550 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:44.306560 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:44.306603 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:44.306809 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.307502 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.309156 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.309731 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.309817 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.309826 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:44.310013 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 19:10:44.310065 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 19:10:44.310191 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:44.306082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:39807 (local address 127.0.203.3:55529)
0620 19:10:44.306171 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:44.306188 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:44.306254 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:44.306478 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:44.306479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:44.306481 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:44.306566 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:44.306568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.307602 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.307603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:44.308961 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:44.308964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.309832 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.309833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:44.309917 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.309919 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:10:44.309990 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:10:44.310062 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":118}
I20260620 19:10:44.310191 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:44.306220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:44409 (local address 127.0.203.3:41763)
0620 19:10:44.306416 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:44.306424 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:44.306494 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:44.306678 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:44.306679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:44.306681 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:44.306739 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:44.306740 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.307602 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.307603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:44.308961 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:44.308964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.309833 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.309833 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:44.309914 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.309919 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:10:44.309989 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 19:10:44.310062 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":47,"thread_start_us":112,"threads_started":1}
I20260620 19:10:44.310542 16316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2f18c8a22440bf8b0b8f46b09cb1c2" candidate_uuid: "d6785bfe80704762a95cc9bf281dd210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e931308008d429cab5bc8a64877424f" is_pre_election: true
I20260620 19:10:44.310724 16316 raft_consensus.cc:2454] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6785bfe80704762a95cc9bf281dd210 in term 0.
I20260620 19:10:44.310698 16450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2f18c8a22440bf8b0b8f46b09cb1c2" candidate_uuid: "d6785bfe80704762a95cc9bf281dd210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" is_pre_election: true
I20260620 19:10:44.310861 16450 raft_consensus.cc:2454] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6785bfe80704762a95cc9bf281dd210 in term 0.
I20260620 19:10:44.311136 16519 leader_election.cc:302] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e931308008d429cab5bc8a64877424f, d6785bfe80704762a95cc9bf281dd210; no voters:
I20260620 19:10:44.311340 16652 raft_consensus.cc:2781] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:44.311419 16652 raft_consensus.cc:482] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:44.311437 16652 raft_consensus.cc:3037] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:44.312165 16652 raft_consensus.cc:504] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } } peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } }
I20260620 19:10:44.312310 16652 leader_election.cc:288] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [CANDIDATE]: Term 1 election: Requested vote from peers 1e931308008d429cab5bc8a64877424f (127.0.203.1:39807), f10fd16e9d0e4285b4c8de19cb7af0e0 (127.0.203.2:44409)
I20260620 19:10:44.312485 16316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2f18c8a22440bf8b0b8f46b09cb1c2" candidate_uuid: "d6785bfe80704762a95cc9bf281dd210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e931308008d429cab5bc8a64877424f"
I20260620 19:10:44.312551 16450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2f18c8a22440bf8b0b8f46b09cb1c2" candidate_uuid: "d6785bfe80704762a95cc9bf281dd210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0"
I20260620 19:10:44.312587 16316 raft_consensus.cc:3037] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:44.312650 16450 raft_consensus.cc:3037] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:44.313310 16450 raft_consensus.cc:2454] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6785bfe80704762a95cc9bf281dd210 in term 1.
I20260620 19:10:44.313316 16316 raft_consensus.cc:2454] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6785bfe80704762a95cc9bf281dd210 in term 1.
I20260620 19:10:44.313465 16519 leader_election.cc:302] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e931308008d429cab5bc8a64877424f, d6785bfe80704762a95cc9bf281dd210; no voters:
I20260620 19:10:44.313572 16652 raft_consensus.cc:2781] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:44.313737 16652 raft_consensus.cc:686] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [term 1 LEADER]: Becoming Leader. State: Replica: d6785bfe80704762a95cc9bf281dd210, State: Running, Role: LEADER
I20260620 19:10:44.313843 16652 consensus_queue.cc:237] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } } peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } }
I20260620 19:10:44.314702 16171 catalog_manager.cc:5495] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6785bfe80704762a95cc9bf281dd210 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "d6785bfe80704762a95cc9bf281dd210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } health_report { overall_health: HEALTHY } } }
I20260620 19:10:44.326874 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:44.324442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51916 (local address 127.0.203.3:43299)
0620 19:10:44.324478 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:44.324480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:44.324489 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:10:44.324576 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:44.324577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:44.324602 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:44.324677 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:44.324836 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.325305 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.326063 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.326269 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.326358 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.326529 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:44.326633 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:44.326728 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:44.326768 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 19:10:44.326788 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:10:44.328747 16450 raft_consensus.cc:1261] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0 [term 1 FOLLOWER]: Refusing update from remote peer d6785bfe80704762a95cc9bf281dd210: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:44.328787 16316 raft_consensus.cc:1261] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Refusing update from remote peer d6785bfe80704762a95cc9bf281dd210: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:44.329103 16654 consensus_queue.cc:1035] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:44.329264 16652 consensus_queue.cc:1035] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:44.331979 16657 mvcc.cc:205] Tried to move back new op lower bound from 7299000911168131072 to 7299000911110623232. Current Snapshot: MvccSnapshot[applied={T|T < 7299000911168131072}]
I20260620 19:10:44.332957 16659 mvcc.cc:205] Tried to move back new op lower bound from 7299000911168131072 to 7299000911110623232. Current Snapshot: MvccSnapshot[applied={T|T < 7299000911168131072 or (T in {7299000911168131072})}]
I20260620 19:10:44.334772 16170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:44.334877 16170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:44.335645 16170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: Sending DeleteTablet for 3 replicas of tablet da2f18c8a22440bf8b0b8f46b09cb1c2
I20260620 19:10:44.336024 16296 tablet_service.cc:1514] Processing DeleteTablet for tablet da2f18c8a22440bf8b0b8f46b09cb1c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:44 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:37178
I20260620 19:10:44.336042 16430 tablet_service.cc:1514] Processing DeleteTablet for tablet da2f18c8a22440bf8b0b8f46b09cb1c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:44 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:53408
I20260620 19:10:44.336061 16564 tablet_service.cc:1514] Processing DeleteTablet for tablet da2f18c8a22440bf8b0b8f46b09cb1c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:44 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:51906
I20260620 19:10:44.336231 16667 tablet_replica.cc:330] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f: stopping tablet replica
I20260620 19:10:44.336319 16667 raft_consensus.cc:2229] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:44.336373 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 16233
I20260620 19:10:44.336440 16667 raft_consensus.cc:2258] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:44.336539 16668 tablet_replica.cc:330] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210: stopping tablet replica
I20260620 19:10:44.336647 16666 tablet_replica.cc:330] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0: stopping tablet replica
I20260620 19:10:44.336773 16668 raft_consensus.cc:2229] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:10:44.336889 16666 raft_consensus.cc:2229] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:44.337033 16666 raft_consensus.cc:2258] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:44.337200 16668 raft_consensus.cc:2258] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:44.337764 16668 ts_tablet_manager.cc:1905] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:44.337764 16666 ts_tablet_manager.cc:1905] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:44.339604 16666 ts_tablet_manager.cc:1918] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:44.339604 16668 ts_tablet_manager.cc:1918] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:44.339675 16668 log.cc:1196] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/da2f18c8a22440bf8b0b8f46b09cb1c2
I20260620 19:10:44.339790 16666 log.cc:1196] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/da2f18c8a22440bf8b0b8f46b09cb1c2
I20260620 19:10:44.339959 16668 ts_tablet_manager.cc:1939] T da2f18c8a22440bf8b0b8f46b09cb1c2 P d6785bfe80704762a95cc9bf281dd210: Deleting consensus metadata
I20260620 19:10:44.340205 16666 ts_tablet_manager.cc:1939] T da2f18c8a22440bf8b0b8f46b09cb1c2 P f10fd16e9d0e4285b4c8de19cb7af0e0: Deleting consensus metadata
I20260620 19:10:44.340618 16157 catalog_manager.cc:4841] TS f10fd16e9d0e4285b4c8de19cb7af0e0 (127.0.203.2:44409): tablet da2f18c8a22440bf8b0b8f46b09cb1c2 (table test-table [id=00dfefb1923e4fbfbb2769ccea5b45a4]) successfully deleted
I20260620 19:10:44.343319 16156 catalog_manager.cc:4841] TS d6785bfe80704762a95cc9bf281dd210 (127.0.203.3:43299): tablet da2f18c8a22440bf8b0b8f46b09cb1c2 (table test-table [id=00dfefb1923e4fbfbb2769ccea5b45a4]) successfully deleted
W20260620 19:10:44.343997 16157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.1:39807 (error 108)
I20260620 19:10:44.344249 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 16367
I20260620 19:10:44.344434 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:44.344308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:39807 (local address 127.0.0.1:37182)
0620 19:10:44.344363 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:44.344396 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.1:39807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 19:10:44.345074 16157 catalog_manager.cc:4584] TS 1e931308008d429cab5bc8a64877424f (127.0.203.1:39807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da2f18c8a22440bf8b0b8f46b09cb1c2: Network error: Client connection negotiation failed: client connection to 127.0.203.1:39807: connect: Connection refused (error 111)
I20260620 19:10:44.349952 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 16501
I20260620 19:10:44.354249 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 16139
I20260620 19:11:04.361044 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:46105
--webserver_interface=127.0.203.62
--webserver_port=34311
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:46105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:04.446588 16670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:04.446794 16670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:04.446816 16670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:04.448033 16670 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 19:11:04.448076 16670 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 19:11:04.448091 16670 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 19:11:04.448102 16670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:11:04.448115 16670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:04.448135 16670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:04.448196 16670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:11:04.448213 16670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:11:04.449700 16670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:46105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:46105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=34311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:04.449983 16670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:04.450235 16670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:04.458846 16679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:04.458894 16670 server_base.cc:1034] running on GCE node
W20260620 19:11:04.458864 16677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:04.458871 16676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:04.459465 16670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:04.459803 16670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:04.460943 16670 hybrid_clock.cc:648] HybridClock initialized: now 1781982664460923 us; error 39 us; skew 500 ppm
Jun 20 19:11:04 dist-test-slave-wd30 krb5kdc[16126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:04.463593 16670 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:11:04.464078 16670 webserver.cc:460] Webserver started at http://127.0.203.62:34311/ using document root <none> and password file <none>
I20260620 19:11:04.464351 16670 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:04.464429 16670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:04.465842 16670 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:11:04.466557 16686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:04.466730 16670 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:11:04.466840 16670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "ece60f07c20b4e4bbbce803100626d68"
format_stamp: "Formatted at 2026-06-20 19:10:43 on dist-test-slave-wd30"
server_key: "b3d36dbbc1da725dd22d46696fad1444"
server_key_iv: "ccd965a1b17a61a273f0d72daa2912e5"
server_key_version: "encryptionkey@0"
I20260620 19:11:04.467321 16670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:04.494057 16670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:04.496436 16670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:04.496645 16670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:04.508249 16670 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:46105
I20260620 19:11:04.508291 16738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:46105 every 8 connection(s)
I20260620 19:11:04.508873 16670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:11:04.511379 16739 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:11:04.512080 16739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: Bootstrap starting.
I20260620 19:11:04.516842 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 16670
I20260620 19:11:04.517199 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:39807
--local_ip_for_outbound_sockets=127.0.203.1
--tserver_master_addrs=127.0.203.62:46105
--webserver_port=38671
--webserver_interface=127.0.203.1
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:11:04.521394 16739 log.cc:824] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:04.523633 16739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:11:04.523972 16739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: Bootstrap complete.
I20260620 19:11:04.526312 16739 raft_consensus.cc:348] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece60f07c20b4e4bbbce803100626d68" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 46105 } }
I20260620 19:11:04.526544 16739 raft_consensus.cc:729] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece60f07c20b4e4bbbce803100626d68, State: Initialized, Role: FOLLOWER
I20260620 19:11:04.526721 16739 consensus_queue.cc:260] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece60f07c20b4e4bbbce803100626d68" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 46105 } }
I20260620 19:11:04.526806 16739 raft_consensus.cc:388] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:11:04.526877 16739 raft_consensus.cc:482] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:11:04.526926 16739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:11:04.528088 16739 raft_consensus.cc:504] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece60f07c20b4e4bbbce803100626d68" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 46105 } }
I20260620 19:11:04.528215 16739 leader_election.cc:302] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ece60f07c20b4e4bbbce803100626d68; no voters:
I20260620 19:11:04.528402 16739 leader_election.cc:288] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:11:04.528446 16743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:11:04.528622 16743 raft_consensus.cc:686] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 2 LEADER]: Becoming Leader. State: Replica: ece60f07c20b4e4bbbce803100626d68, State: Running, Role: LEADER
I20260620 19:11:04.528715 16739 sys_catalog.cc:564] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:11:04.528771 16743 consensus_queue.cc:237] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece60f07c20b4e4bbbce803100626d68" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 46105 } }
I20260620 19:11:04.529078 16743 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ece60f07c20b4e4bbbce803100626d68. Latest consensus state: current_term: 2 leader_uuid: "ece60f07c20b4e4bbbce803100626d68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece60f07c20b4e4bbbce803100626d68" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 46105 } } }
I20260620 19:11:04.529173 16743 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:04.529081 16744 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ece60f07c20b4e4bbbce803100626d68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece60f07c20b4e4bbbce803100626d68" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 46105 } } }
I20260620 19:11:04.529331 16744 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:04.530714 16759 catalog_manager.cc:1259] Loaded cluster ID: 7e1dbb4029a64b3385c7d43ee0611b2e
I20260620 19:11:04.530826 16759 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: loading cluster ID for follower catalog manager: success
I20260620 19:11:04.532028 16759 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: acquiring CA information for follower catalog manager: success
I20260620 19:11:04.532471 16759 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:11:04.532555 16749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:11:04.532968 16749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:11:04.533078 16749 catalog_manager.cc:1259] Loaded cluster ID: 7e1dbb4029a64b3385c7d43ee0611b2e
I20260620 19:11:04.533113 16749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:11:04.533273 16749 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:11:04.533357 16749 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: Loaded TSK: 0
I20260620 19:11:04.544387 16749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: Generated new TSK 1
I20260620 19:11:04.544512 16749 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:11:04.602416 16741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:04.602653 16741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:04.602695 16741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:04.603964 16741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:04.604034 16741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:04.604081 16741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:11:04.605580 16741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:39807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=38671
--tserver_master_addrs=127.0.203.62:46105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:04.605850 16741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:04.606097 16741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:04.611236 16766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:04.611194 16767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:04.611387 16741 server_base.cc:1034] running on GCE node
W20260620 19:11:04.611191 16769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:04.611696 16741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:04.611970 16741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:04.613157 16741 hybrid_clock.cc:648] HybridClock initialized: now 1781982664613145 us; error 43 us; skew 500 ppm
Jun 20 19:11:04 dist-test-slave-wd30 krb5kdc[16126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:04.616117 16741 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:11:04.616546 16741 webserver.cc:460] Webserver started at http://127.0.203.1:38671/ using document root <none> and password file <none>
I20260620 19:11:04.616796 16741 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:04.616883 16741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:04.618328 16741 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:11:04.619077 16776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:04.619259 16741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 19:11:04.619364 16741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "1e931308008d429cab5bc8a64877424f"
format_stamp: "Formatted at 2026-06-20 19:10:43 on dist-test-slave-wd30"
server_key: "00dfdc59af68ef5c9eb22746ffd4cccd"
server_key_iv: "ba4ba3634e53a0a0dbd1cb72d1f0175c"
server_key_version: "encryptionkey@0"
I20260620 19:11:04.619733 16741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:04.648233 16741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:04.650715 16741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:04.650898 16741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:04.651226 16741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:04.651767 16783 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:11:04.653460 16741 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:11:04.653514 16741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 19:11:04.653551 16741 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:11:04.654263 16741 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:11:04.654311 16741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 19:11:04.654358 16783 tablet_bootstrap.cc:492] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f: Bootstrap starting.
I20260620 19:11:04.662410 16741 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:39807
I20260620 19:11:04.662482 16890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:39807 every 8 connection(s)
I20260620 19:11:04.663195 16741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:11:04.665982 16783 log.cc:824] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:04.668565 16783 tablet_bootstrap.cc:492] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:11:04.668921 16783 tablet_bootstrap.cc:492] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f: Bootstrap complete.
I20260620 19:11:04.670006 16783 ts_tablet_manager.cc:1397] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260620 19:11:04.671456 16783 raft_consensus.cc:348] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } } peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } }
I20260620 19:11:04.671741 16783 raft_consensus.cc:729] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e931308008d429cab5bc8a64877424f, State: Initialized, Role: FOLLOWER
I20260620 19:11:04.671921 16783 consensus_queue.cc:260] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } } peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } }
I20260620 19:11:04.672246 16783 ts_tablet_manager.cc:1428] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:11:04.672961 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 16741
Jun 20 19:11:04 dist-test-slave-wd30 krb5kdc[16126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:11:04.673264 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:44409
--local_ip_for_outbound_sockets=127.0.203.2
--tserver_master_addrs=127.0.203.62:46105
--webserver_port=40031
--webserver_interface=127.0.203.2
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:11:04.677704 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:04.664120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:34641 (local address 127.0.203.62:46105)
0620 19:11:04.664352 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:04.664362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:04.666425 (+ 2063us) server_negotiation.cc:407] Connection header received
0620 19:11:04.667064 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:04.667072 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:04.667186 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:04.667357 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:04.668012 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.669442 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.670609 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.671300 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.671414 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:04.673806 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:04.673810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:04.673811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:04.673825 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:04.675817 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:04.676305 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:04.676309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:04.676309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:04.676416 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:04.676705 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:04.676706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:04.676707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:04.676955 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:04.677047 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:04.677362 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 19:11:04.677496 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":83,"threads_started":1}
I20260620 19:11:04.677717 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:04.664055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.1:34641)
0620 19:11:04.666203 (+ 2148us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:04.666255 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:04.666825 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:04.667373 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:04.667377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:04.667668 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:04.667876 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:04.667882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.669567 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.669568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:04.670528 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:04.670530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.671386 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.671387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:04.671488 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:04.672078 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:11:04.672088 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:04.673682 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:04.675915 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:04.675927 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:04.675929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:04.676173 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:04.676499 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:04.676500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:04.676501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:04.676623 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:04.677037 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:04.677045 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:04.677189 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 19:11:04.677302 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 19:11:04.677493 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2099,"thread_start_us":116,"threads_started":1}
I20260620 19:11:04.678663 16891 heartbeater.cc:347] Connected to a master server at 127.0.203.62:46105
I20260620 19:11:04.678766 16891 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:04.679186 16891 heartbeater.cc:511] Master 127.0.203.62:46105 requested a full tablet report, sending...
I20260620 19:11:04.680092 16703 ts_manager.cc:194] Registered new tserver with Master: 1e931308008d429cab5bc8a64877424f (127.0.203.1:39807)
I20260620 19:11:04.681542 16703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:34641
I20260620 19:11:04.682513 16891 heartbeater.cc:503] Master 127.0.203.62:46105 was elected leader, sending a full tablet report...
I20260620 19:11:04.686915 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:04.680829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:39807 (local address 127.0.0.1:37804)
0620 19:11:04.681019 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:04.681056 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:04.681128 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:04.683447 (+ 2319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:04.683450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:04.683452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:04.683516 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:04.683517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.684905 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.684906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:04.685736 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:04.685738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.686645 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.686646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:04.686722 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:04.686733 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 19:11:04.686801 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 19:11:04.686851 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":119,"threads_started":1}
I20260620 19:11:04.686942 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:04.681080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37804 (local address 127.0.203.1:39807)
0620 19:11:04.683248 (+ 2168us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:04.683253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:04.683267 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:11:04.683310 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:04.683312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:04.683331 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:04.683436 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:04.683598 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.684805 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.685820 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.686544 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.686620 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:04.686686 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:04.686830 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 19:11:04.686869 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2125,"thread_start_us":110,"threads_started":1}
I20260620 19:11:04.687615 16825 tablet_service.cc:1514] Processing DeleteTablet for tablet da2f18c8a22440bf8b0b8f46b09cb1c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:44 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:37804
I20260620 19:11:04.688195 16900 tablet_replica.cc:330] stopping tablet replica
I20260620 19:11:04.688318 16900 raft_consensus.cc:2229] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:11:04.688391 16900 raft_consensus.cc:2258] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:11:04.688699 16900 ts_tablet_manager.cc:1905] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:11:04.690852 16900 ts_tablet_manager.cc:1918] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:11:04.690917 16900 log.cc:1196] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/wal/wals/da2f18c8a22440bf8b0b8f46b09cb1c2
I20260620 19:11:04.691211 16900 ts_tablet_manager.cc:1939] T da2f18c8a22440bf8b0b8f46b09cb1c2 P 1e931308008d429cab5bc8a64877424f: Deleting consensus metadata
I20260620 19:11:04.691752 16689 catalog_manager.cc:4841] TS 1e931308008d429cab5bc8a64877424f (127.0.203.1:39807): tablet da2f18c8a22440bf8b0b8f46b09cb1c2 (table test-table [id=00dfefb1923e4fbfbb2769ccea5b45a4]) successfully deleted
W20260620 19:11:04.759068 16897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:04.759349 16897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:04.759392 16897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:04.760602 16897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:04.760676 16897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:04.760723 16897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:11:04.762256 16897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:44409
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=40031
--tserver_master_addrs=127.0.203.62:46105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:04.762537 16897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:04.762781 16897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:04.767851 16906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:04.767895 16905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:04.768016 16897 server_base.cc:1034] running on GCE node
W20260620 19:11:04.767866 16908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:04.768306 16897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:04.768544 16897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:04.769727 16897 hybrid_clock.cc:648] HybridClock initialized: now 1781982664769693 us; error 55 us; skew 500 ppm
Jun 20 19:11:04 dist-test-slave-wd30 krb5kdc[16126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:04.772481 16897 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:11:04.772924 16897 webserver.cc:460] Webserver started at http://127.0.203.2:40031/ using document root <none> and password file <none>
I20260620 19:11:04.773156 16897 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:04.773238 16897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:04.774636 16897 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:11:04.778749 16915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:04.778959 16897 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 19:11:04.779078 16897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0"
format_stamp: "Formatted at 2026-06-20 19:10:43 on dist-test-slave-wd30"
server_key: "62684ef19fc6df101406cbde57e82130"
server_key_iv: "779c3f08b3e82327c91ffdf55e4f0a85"
server_key_version: "encryptionkey@0"
I20260620 19:11:04.779601 16897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:04.795591 16897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:04.798254 16897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:04.798432 16897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:04.798683 16897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:04.799010 16897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:04.799067 16897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:04.799167 16897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:04.799207 16897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:04.805498 16897 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:44409
I20260620 19:11:04.805541 17028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:44409 every 8 connection(s)
I20260620 19:11:04.805920 16897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:11:04.809170 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 16897
I20260620 19:11:04.809481 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:43299
--local_ip_for_outbound_sockets=127.0.203.3
--tserver_master_addrs=127.0.203.62:46105
--webserver_port=43667
--webserver_interface=127.0.203.3
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:11:04 dist-test-slave-wd30 krb5kdc[16126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:11:04.817552 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:04.807005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:45621 (local address 127.0.203.62:46105)
0620 19:11:04.807046 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:04.807048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:04.807060 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:11:04.807844 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:04.807846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:04.807860 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:04.807901 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:04.809351 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.810245 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.811966 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.812180 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.812283 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:04.814511 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:04.814513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:04.814513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:04.814522 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:04.816079 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:04.816509 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:04.816510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:04.816511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:04.816577 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:04.816867 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:04.816868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:04.816868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:04.816992 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:04.817065 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:04.817360 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 19:11:04.817458 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:11:04.817754 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:04.806726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.2:45621)
0620 19:11:04.806895 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:04.806946 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:04.807741 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:04.807993 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:04.807998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:04.808857 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:04.809202 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:04.809210 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.810381 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.810382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:04.811872 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:04.811875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.812292 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.812294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:04.812440 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:04.812997 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:11:04.813006 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:04.814406 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:04.816183 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:04.816191 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:04.816193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:04.816424 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:04.816652 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:04.816653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:04.816654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:04.816773 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:04.817067 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:04.817074 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:04.817225 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 19:11:04.817339 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 19:11:04.817505 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260620 19:11:04.818374 17029 heartbeater.cc:347] Connected to a master server at 127.0.203.62:46105
I20260620 19:11:04.818495 17029 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:04.818710 17029 heartbeater.cc:511] Master 127.0.203.62:46105 requested a full tablet report, sending...
I20260620 19:11:04.819101 16703 ts_manager.cc:194] Registered new tserver with Master: f10fd16e9d0e4285b4c8de19cb7af0e0 (127.0.203.2:44409)
I20260620 19:11:04.819531 16703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:45621
W20260620 19:11:04.905891 17032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:04.906119 17032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:04.906143 17032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:04.907562 17032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:04.907611 17032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:04.907639 17032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:11:04.909123 17032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:43299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=43667
--tserver_master_addrs=127.0.203.62:46105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:04.909394 17032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:04.909644 17032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:04.914776 17037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:04.914773 17038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:04.914839 17040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:04.915079 17032 server_base.cc:1034] running on GCE node
I20260620 19:11:04.915297 17032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:04.915565 17032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:04.916731 17032 hybrid_clock.cc:648] HybridClock initialized: now 1781982664916718 us; error 33 us; skew 500 ppm
Jun 20 19:11:04 dist-test-slave-wd30 krb5kdc[16126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:04.919428 17032 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:11:04.919847 17032 webserver.cc:460] Webserver started at http://127.0.203.3:43667/ using document root <none> and password file <none>
I20260620 19:11:04.920056 17032 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:04.920114 17032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:04.921480 17032 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:11:04.922107 17047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:04.922276 17032 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:11:04.922376 17032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "d6785bfe80704762a95cc9bf281dd210"
format_stamp: "Formatted at 2026-06-20 19:10:44 on dist-test-slave-wd30"
server_key: "136c180a87920cc264ae1ea137fd76bd"
server_key_iv: "7971f97415af26e7935a35e03698bb74"
server_key_version: "encryptionkey@0"
I20260620 19:11:04.922817 17032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:04.936218 17032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:04.938800 17032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:04.938985 17032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:04.939314 17032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:04.939651 17032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:04.939711 17032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:04.939759 17032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:04.939795 17032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:04.946143 17032 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:43299
I20260620 19:11:04.946184 17160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:43299 every 8 connection(s)
I20260620 19:11:04.946794 17032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:11:04 dist-test-slave-wd30 krb5kdc[16126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:11:04.955793 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 17032
I20260620 19:11:04.957154 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:04.947639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:47319 (local address 127.0.203.62:46105)
0620 19:11:04.947698 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:04.947701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:04.947991 (+ 290us) server_negotiation.cc:407] Connection header received
0620 19:11:04.948843 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:04.948845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:04.948860 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:04.948929 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:04.950019 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.950530 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.951630 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.951824 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.951893 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:04.953927 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:04.953929 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:04.953929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:04.953936 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:04.955527 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:04.956043 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:04.956045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:04.956045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:04.956112 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:04.956400 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:04.956401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:04.956402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:04.956579 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:04.956651 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:04.957017 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 19:11:04.957088 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:11:04.957445 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:04.947627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.3:47319)
0620 19:11:04.947815 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:04.947867 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:04.948709 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:04.949075 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:04.949081 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:04.949693 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:04.949900 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:04.949904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.950620 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.950621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:04.951536 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:04.951538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.951905 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.951905 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:04.952001 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:04.952492 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:11:04.952500 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:04.953843 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:04.955604 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:04.955611 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:04.955613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:04.955942 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:04.956193 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:04.956194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:04.956194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:04.956319 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:04.956655 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:04.956661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:04.956852 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 19:11:04.956979 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 19:11:04.957185 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
I20260620 19:11:04.957931 17161 heartbeater.cc:347] Connected to a master server at 127.0.203.62:46105
I20260620 19:11:04.958317 17161 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:04.958531 17161 heartbeater.cc:511] Master 127.0.203.62:46105 requested a full tablet report, sending...
I20260620 19:11:04.959034 16703 ts_manager.cc:194] Registered new tserver with Master: d6785bfe80704762a95cc9bf281dd210 (127.0.203.3:43299)
I20260620 19:11:04.959705 16703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:47319
I20260620 19:11:04.963717 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:04.956266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.0.203.62:46105)
0620 19:11:04.956427 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:04.956429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:04.956514 (+ 85us) server_negotiation.cc:407] Connection header received
0620 19:11:04.956555 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:04.956556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:04.956570 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:04.956642 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:04.957479 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.958123 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.959276 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.959569 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.959645 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:04.961244 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:04.961245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:04.961246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:04.961253 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:04.962632 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:04.962973 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:04.962974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:04.962975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:04.963029 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:04.963258 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:04.963259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:04.963259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:04.963359 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:04.963499 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:04.963529 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 19:11:04.963593 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":73,"threads_started":1}
I20260620 19:11:04.964604 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:11:04.967356 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:04.964962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.0.203.62:46105)
0620 19:11:04.965019 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:04.965020 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:04.965137 (+ 117us) server_negotiation.cc:407] Connection header received
0620 19:11:04.965204 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:04.965204 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:04.965233 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:04.965290 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:11:04.965377 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.965830 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.966579 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.966772 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.966821 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:04.966968 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:11:04.967059 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:11:04.967156 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:04.967256 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 19:11:04.967283 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:11:04.967655 16703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:11:04.969314 16703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:11:04.971642 16825 tablet_service.cc:1467] Processing CreateTablet for tablet bdd43259792b40b3a32df8437cb466c3 (DEFAULT_TABLE table=test-table [id=ef0583e42b1746229462671140f0c6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:04.971863 16825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdd43259792b40b3a32df8437cb466c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:04.973587 16783 tablet_bootstrap.cc:492] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f: Bootstrap starting.
I20260620 19:11:04.973943 16783 tablet_bootstrap.cc:654] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:04.974473 16783 tablet_bootstrap.cc:492] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f: No bootstrap required, opened a new log
I20260620 19:11:04.974539 16783 ts_tablet_manager.cc:1397] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:11:04.974660 16783 raft_consensus.cc:348] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } } peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } }
I20260620 19:11:04.974745 16783 raft_consensus.cc:374] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:04.974800 16783 raft_consensus.cc:729] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e931308008d429cab5bc8a64877424f, State: Initialized, Role: FOLLOWER
I20260620 19:11:04.974857 16783 consensus_queue.cc:260] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } } peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } }
I20260620 19:11:04.974954 16783 ts_tablet_manager.cc:1428] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:11:04.975617 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:04.971208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:44409 (local address 127.0.0.1:39376)
0620 19:11:04.971519 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:04.971531 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:04.971582 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:04.971907 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:04.971908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:04.971910 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:04.971982 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:04.971983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.973271 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.973273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:04.974300 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:04.974303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.975355 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.975357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:04.975424 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:04.975427 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:11:04.975495 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 19:11:04.975537 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":26,"thread_start_us":70,"threads_started":1}
I20260620 19:11:04.975871 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:04.971426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39376 (local address 127.0.203.2:44409)
0620 19:11:04.971615 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:04.971619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:04.971627 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:11:04.971664 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:04.971666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:04.971680 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:04.971823 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:04.972081 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.973160 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.974397 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.975177 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.975253 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:04.975315 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:04.975735 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 19:11:04.975780 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
I20260620 19:11:04.976096 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:04.971208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:43299 (local address 127.0.0.1:51640)
0620 19:11:04.971265 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:04.971278 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:04.971326 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:04.971708 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:04.971709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:04.971710 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:04.971754 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:04.971755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.972834 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.972835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:04.973648 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:04.973651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.975936 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.975936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:04.975984 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:04.975986 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:11:04.976021 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 19:11:04.976054 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 19:11:04.976176 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:04.971319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.0.203.3:43299)
0620 19:11:04.971500 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:04.971505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:04.971515 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:11:04.971554 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:04.971557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:04.971575 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:04.971713 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:04.971806 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.972754 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.974189 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:04.975844 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:04.975938 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:04.976008 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:04.976040 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 19:11:04.976081 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
I20260620 19:11:04.976655 16963 tablet_service.cc:1467] Processing CreateTablet for tablet bdd43259792b40b3a32df8437cb466c3 (DEFAULT_TABLE table=test-table [id=ef0583e42b1746229462671140f0c6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:04.976851 17095 tablet_service.cc:1467] Processing CreateTablet for tablet bdd43259792b40b3a32df8437cb466c3 (DEFAULT_TABLE table=test-table [id=ef0583e42b1746229462671140f0c6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:04.977051 16963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdd43259792b40b3a32df8437cb466c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:04.977222 17095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdd43259792b40b3a32df8437cb466c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:04.979872 17171 tablet_bootstrap.cc:492] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0: Bootstrap starting.
I20260620 19:11:04.980073 17172 tablet_bootstrap.cc:492] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210: Bootstrap starting.
I20260620 19:11:04.980568 17171 tablet_bootstrap.cc:654] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:04.980716 17172 tablet_bootstrap.cc:654] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:04.981117 17171 log.cc:824] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:04.981266 17172 log.cc:824] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:04.981940 17171 tablet_bootstrap.cc:492] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0: No bootstrap required, opened a new log
I20260620 19:11:04.982039 17172 tablet_bootstrap.cc:492] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210: No bootstrap required, opened a new log
I20260620 19:11:04.982030 17171 ts_tablet_manager.cc:1397] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:11:04.982106 17172 ts_tablet_manager.cc:1397] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:11:04.984189 17171 raft_consensus.cc:348] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } } peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } }
I20260620 19:11:04.984189 17172 raft_consensus.cc:348] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } } peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } }
I20260620 19:11:04.984392 17172 raft_consensus.cc:374] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:04.984392 17171 raft_consensus.cc:374] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:04.984433 17172 raft_consensus.cc:729] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6785bfe80704762a95cc9bf281dd210, State: Initialized, Role: FOLLOWER
I20260620 19:11:04.984433 17171 raft_consensus.cc:729] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f10fd16e9d0e4285b4c8de19cb7af0e0, State: Initialized, Role: FOLLOWER
I20260620 19:11:04.984565 17171 consensus_queue.cc:260] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } } peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } }
I20260620 19:11:04.984565 17172 consensus_queue.cc:260] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } } peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } }
I20260620 19:11:04.984871 17171 ts_tablet_manager.cc:1428] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:11:04.984871 17172 ts_tablet_manager.cc:1428] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:11:04.984892 17161 heartbeater.cc:503] Master 127.0.203.62:46105 was elected leader, sending a full tablet report...
I20260620 19:11:04.985158 17029 heartbeater.cc:503] Master 127.0.203.62:46105 was elected leader, sending a full tablet report...
W20260620 19:11:05.056617 17030 tablet.cc:2369] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:11:05.154584 16896 raft_consensus.cc:482] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:11:05.154716 16896 raft_consensus.cc:504] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } } peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } }
I20260620 19:11:05.155108 16896 leader_election.cc:288] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f10fd16e9d0e4285b4c8de19cb7af0e0 (127.0.203.2:44409), d6785bfe80704762a95cc9bf281dd210 (127.0.203.3:43299)
I20260620 19:11:05.158318 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:05.155396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:53975 (local address 127.0.203.2:44409)
0620 19:11:05.155449 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:05.155451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:05.155460 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:11:05.155501 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:05.155502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:05.155515 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:05.155588 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:05.155771 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:05.156443 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:05.157498 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:05.158020 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:05.158072 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:05.158080 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:05.158239 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 19:11:05.158266 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 19:11:05.158353 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:05.155291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:44409 (local address 127.0.203.1:53975)
0620 19:11:05.155360 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:05.155373 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:05.155420 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:05.155630 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:05.155631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:05.155633 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:05.155696 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:05.155698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:05.156518 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:05.156519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:05.157394 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:05.157396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:05.158092 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:05.158093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:05.158165 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:05.158170 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:11:05.158226 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 19:11:05.158266 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 19:11:05.158717 16983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdd43259792b40b3a32df8437cb466c3" candidate_uuid: "1e931308008d429cab5bc8a64877424f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" is_pre_election: true
I20260620 19:11:05.158807 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:05.155441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:33215 (local address 127.0.203.3:43299)
0620 19:11:05.155502 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:05.155505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:05.155579 (+ 74us) server_negotiation.cc:407] Connection header received
0620 19:11:05.155658 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:05.155659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:05.155671 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:05.155732 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:05.156257 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:05.156858 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:05.157895 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:05.158412 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:05.158479 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:05.158511 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:05.158705 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 19:11:05.158740 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:11:05.158823 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:05.155368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:43299 (local address 127.0.203.1:33215)
0620 19:11:05.155511 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:05.155522 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:05.155579 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:05.155769 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:05.155769 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:05.155771 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:05.156170 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:05.156172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:05.156942 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:05.156943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:05.157809 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:05.157811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:05.158551 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:05.158552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:05.158622 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:05.158625 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:11:05.158700 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 19:11:05.158738 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260620 19:11:05.158882 16983 raft_consensus.cc:2454] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e931308008d429cab5bc8a64877424f in term 0.
I20260620 19:11:05.159091 17115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdd43259792b40b3a32df8437cb466c3" candidate_uuid: "1e931308008d429cab5bc8a64877424f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6785bfe80704762a95cc9bf281dd210" is_pre_election: true
I20260620 19:11:05.159235 16779 leader_election.cc:302] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e931308008d429cab5bc8a64877424f, f10fd16e9d0e4285b4c8de19cb7af0e0; no voters:
I20260620 19:11:05.159292 17115 raft_consensus.cc:2454] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e931308008d429cab5bc8a64877424f in term 0.
I20260620 19:11:05.159360 16896 raft_consensus.cc:2781] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:11:05.159412 16896 raft_consensus.cc:482] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:11:05.159428 16896 raft_consensus.cc:3037] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:05.160163 16896 raft_consensus.cc:504] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } } peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } }
I20260620 19:11:05.160293 16896 leader_election.cc:288] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [CANDIDATE]: Term 1 election: Requested vote from peers f10fd16e9d0e4285b4c8de19cb7af0e0 (127.0.203.2:44409), d6785bfe80704762a95cc9bf281dd210 (127.0.203.3:43299)
I20260620 19:11:05.160449 17115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdd43259792b40b3a32df8437cb466c3" candidate_uuid: "1e931308008d429cab5bc8a64877424f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6785bfe80704762a95cc9bf281dd210"
I20260620 19:11:05.160481 16983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdd43259792b40b3a32df8437cb466c3" candidate_uuid: "1e931308008d429cab5bc8a64877424f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0"
I20260620 19:11:05.160545 17115 raft_consensus.cc:3037] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:05.160550 16983 raft_consensus.cc:3037] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:05.161437 16983 raft_consensus.cc:2454] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e931308008d429cab5bc8a64877424f in term 1.
I20260620 19:11:05.161446 17115 raft_consensus.cc:2454] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e931308008d429cab5bc8a64877424f in term 1.
I20260620 19:11:05.161604 16779 leader_election.cc:302] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e931308008d429cab5bc8a64877424f, f10fd16e9d0e4285b4c8de19cb7af0e0; no voters:
I20260620 19:11:05.161708 16896 raft_consensus.cc:2781] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:05.161888 16896 raft_consensus.cc:686] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 1 LEADER]: Becoming Leader. State: Replica: 1e931308008d429cab5bc8a64877424f, State: Running, Role: LEADER
I20260620 19:11:05.161953 16896 consensus_queue.cc:237] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } } peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } }
I20260620 19:11:05.162729 16703 catalog_manager.cc:5495] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e931308008d429cab5bc8a64877424f (127.0.203.1). New cstate: current_term: 1 leader_uuid: "1e931308008d429cab5bc8a64877424f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } health_report { overall_health: HEALTHY } } }
W20260620 19:11:05.164242 16892 tablet.cc:2369] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:11:05.197522 17162 tablet.cc:2369] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:11:05.202785 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:05.200325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37820 (local address 127.0.203.1:39807)
0620 19:11:05.200457 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:05.200459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:05.200466 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:11:05.200504 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:05.200505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:05.200523 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:05.200613 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:11:05.200752 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:05.201241 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:05.202010 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:05.202210 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:05.202296 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:05.202464 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:11:05.202523 (+ 59us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 19:11:05.202677 (+ 154us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37820: token expired
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":54,"threads_started":1}
W20260620 19:11:05.202859 17181 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37820: token expired
I20260620 19:11:05.211025 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:05.203391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.0.203.62:46105)
0620 19:11:05.203461 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:05.203463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:05.203470 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:11:05.203491 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:05.203492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:05.203508 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:05.203572 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:05.204069 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:05.204801 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:05.205860 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:05.206248 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:05.206338 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:05.207915 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:05.207916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:05.207916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:05.207923 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:05.209855 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:05.210204 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:05.210205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:05.210205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:05.210255 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:05.210424 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:05.210425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:05.210425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:05.210576 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:05.210652 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:05.210740 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 19:11:05.210859 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:11:05.214093 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:05.212005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37830 (local address 127.0.203.1:39807)
0620 19:11:05.212072 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:05.212074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:05.212092 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:11:05.212116 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:05.212116 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:05.212126 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:05.212163 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:11:05.212255 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:05.212706 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:05.213399 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:05.213595 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:05.213658 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:05.213763 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:11:05.213858 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:11:05.213896 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:05.213947 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 19:11:05.213965 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:11:05.215696 16983 raft_consensus.cc:1261] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0 [term 1 FOLLOWER]: Refusing update from remote peer 1e931308008d429cab5bc8a64877424f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:11:05.215701 17115 raft_consensus.cc:1261] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210 [term 1 FOLLOWER]: Refusing update from remote peer 1e931308008d429cab5bc8a64877424f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:11:05.216027 16896 consensus_queue.cc:1035] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:11:05.216144 17178 consensus_queue.cc:1035] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:11:05.218927 17183 mvcc.cc:205] Tried to move back new op lower bound from 7299000996720730112 to 7299000996504109056. Current Snapshot: MvccSnapshot[applied={T|T < 7299000996720730112}]
I20260620 19:11:05.220852 16703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:11:05.220970 16703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:11:05.222064 16703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: Sending DeleteTablet for 3 replicas of tablet bdd43259792b40b3a32df8437cb466c3
I20260620 19:11:05.222538 16963 tablet_service.cc:1514] Processing DeleteTablet for tablet bdd43259792b40b3a32df8437cb466c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:05 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:39376
I20260620 19:11:05.222592 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 16741
I20260620 19:11:05.222580 17095 tablet_service.cc:1514] Processing DeleteTablet for tablet bdd43259792b40b3a32df8437cb466c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:05 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:51640
I20260620 19:11:05.223383 17193 tablet_replica.cc:330] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210: stopping tablet replica
I20260620 19:11:05.223547 17192 tablet_replica.cc:330] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0: stopping tablet replica
I20260620 19:11:05.223556 17193 raft_consensus.cc:2229] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:11:05.223654 17192 raft_consensus.cc:2229] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:11:05.223693 17193 raft_consensus.cc:2258] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:11:05.223748 17192 raft_consensus.cc:2258] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:11:05.224239 17192 ts_tablet_manager.cc:1905] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:11:05.224239 17193 ts_tablet_manager.cc:1905] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:11:05.225958 17192 ts_tablet_manager.cc:1918] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:11:05.226001 17193 ts_tablet_manager.cc:1918] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:11:05.226039 17192 log.cc:1196] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/bdd43259792b40b3a32df8437cb466c3
I20260620 19:11:05.226061 17193 log.cc:1196] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/bdd43259792b40b3a32df8437cb466c3
I20260620 19:11:05.226356 17192 ts_tablet_manager.cc:1939] T bdd43259792b40b3a32df8437cb466c3 P f10fd16e9d0e4285b4c8de19cb7af0e0: Deleting consensus metadata
I20260620 19:11:05.226368 17193 ts_tablet_manager.cc:1939] T bdd43259792b40b3a32df8437cb466c3 P d6785bfe80704762a95cc9bf281dd210: Deleting consensus metadata
I20260620 19:11:05.226825 16689 catalog_manager.cc:4841] TS f10fd16e9d0e4285b4c8de19cb7af0e0 (127.0.203.2:44409): tablet bdd43259792b40b3a32df8437cb466c3 (table test-table [id=ef0583e42b1746229462671140f0c6f3]) successfully deleted
I20260620 19:11:05.227015 16688 catalog_manager.cc:4841] TS d6785bfe80704762a95cc9bf281dd210 (127.0.203.3:43299): tablet bdd43259792b40b3a32df8437cb466c3 (table test-table [id=ef0583e42b1746229462671140f0c6f3]) successfully deleted
W20260620 19:11:05.228942 16689 connection.cc:535] client connection to 127.0.203.1:39807 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 19:11:05.229055 16689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:11:05.229048 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 16897
I20260620 19:11:05.230029 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:05.229872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:39807 (local address 127.0.0.1:37836)
0620 19:11:05.229937 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:05.229983 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.1:39807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 19:11:05.230155 16689 catalog_manager.cc:4584] TS 1e931308008d429cab5bc8a64877424f (127.0.203.1:39807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bdd43259792b40b3a32df8437cb466c3: Network error: Client connection negotiation failed: client connection to 127.0.203.1:39807: connect: Connection refused (error 111)
I20260620 19:11:05.234457 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 17032
I20260620 19:11:05.238927 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 16670
I20260620 19:11:25.244544 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:46105
--webserver_interface=127.0.203.62
--webserver_port=34311
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:46105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:25.329872 17195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:25.330129 17195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:25.330154 17195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:25.331403 17195 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 19:11:25.331456 17195 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 19:11:25.331471 17195 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 19:11:25.331485 17195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:11:25.331499 17195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:25.331513 17195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:25.331525 17195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:11:25.331537 17195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:11:25.333081 17195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:46105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:46105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=34311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:25.333303 17195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:25.333500 17195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:25.338994 17201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:25.338999 17200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:25.339013 17195 server_base.cc:1034] running on GCE node
W20260620 19:11:25.338994 17203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:25.339534 17195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:25.339848 17195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:25.341010 17195 hybrid_clock.cc:648] HybridClock initialized: now 1781982685341003 us; error 29 us; skew 500 ppm
Jun 20 19:11:25 dist-test-slave-wd30 krb5kdc[16126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:25.343416 17195 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:11:25.343869 17195 webserver.cc:460] Webserver started at http://127.0.203.62:34311/ using document root <none> and password file <none>
I20260620 19:11:25.344091 17195 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:25.344159 17195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:25.345486 17195 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:11:25.346130 17210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:25.346298 17195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:11:25.346410 17195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "ece60f07c20b4e4bbbce803100626d68"
format_stamp: "Formatted at 2026-06-20 19:10:43 on dist-test-slave-wd30"
server_key: "b3d36dbbc1da725dd22d46696fad1444"
server_key_iv: "ccd965a1b17a61a273f0d72daa2912e5"
server_key_version: "encryptionkey@0"
I20260620 19:11:25.346751 17195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:25.358946 17195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:25.360954 17195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:25.361063 17195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:25.364831 17195 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:46105
I20260620 19:11:25.364872 17262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:46105 every 8 connection(s)
I20260620 19:11:25.365257 17195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:11:25.366789 17263 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:11:25.367501 17263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: Bootstrap starting.
I20260620 19:11:25.369709 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 17195
I20260620 19:11:25.370038 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:39807
--local_ip_for_outbound_sockets=127.0.203.1
--tserver_master_addrs=127.0.203.62:46105
--webserver_port=38671
--webserver_interface=127.0.203.1
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:11:25.377214 17263 log.cc:824] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:25.379765 17263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:11:25.380149 17263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: Bootstrap complete.
I20260620 19:11:25.382525 17263 raft_consensus.cc:348] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece60f07c20b4e4bbbce803100626d68" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 46105 } }
I20260620 19:11:25.382805 17263 raft_consensus.cc:729] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece60f07c20b4e4bbbce803100626d68, State: Initialized, Role: FOLLOWER
I20260620 19:11:25.383034 17263 consensus_queue.cc:260] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece60f07c20b4e4bbbce803100626d68" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 46105 } }
I20260620 19:11:25.383153 17263 raft_consensus.cc:388] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:11:25.383213 17263 raft_consensus.cc:482] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:11:25.383273 17263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:11:25.384510 17263 raft_consensus.cc:504] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece60f07c20b4e4bbbce803100626d68" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 46105 } }
I20260620 19:11:25.384639 17263 leader_election.cc:302] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ece60f07c20b4e4bbbce803100626d68; no voters:
I20260620 19:11:25.384843 17263 leader_election.cc:288] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 19:11:25.384883 17267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:11:25.385149 17267 raft_consensus.cc:686] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [term 3 LEADER]: Becoming Leader. State: Replica: ece60f07c20b4e4bbbce803100626d68, State: Running, Role: LEADER
I20260620 19:11:25.385320 17263 sys_catalog.cc:564] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:11:25.385268 17267 consensus_queue.cc:237] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece60f07c20b4e4bbbce803100626d68" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 46105 } }
I20260620 19:11:25.385628 17267 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ece60f07c20b4e4bbbce803100626d68. Latest consensus state: current_term: 3 leader_uuid: "ece60f07c20b4e4bbbce803100626d68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece60f07c20b4e4bbbce803100626d68" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 46105 } } }
I20260620 19:11:25.385700 17267 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:25.386232 17268 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ece60f07c20b4e4bbbce803100626d68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece60f07c20b4e4bbbce803100626d68" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 46105 } } }
I20260620 19:11:25.386338 17268 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68 [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:25.386478 17272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:11:25.387600 17272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:11:25.387718 17272 catalog_manager.cc:1259] Loaded cluster ID: 7e1dbb4029a64b3385c7d43ee0611b2e
I20260620 19:11:25.387967 17272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:11:25.389129 17272 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:11:25.389323 17272 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: Loaded TSK: 0
I20260620 19:11:25.389374 17272 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ece60f07c20b4e4bbbce803100626d68: Loaded TSK: 1
F20260620 19:11:25.389621 17272 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe85e06b9ad google::LogMessage::Fail() at ??:0
@ 0x7fe85e06e88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe85e06b4de google::LogMessage::Flush() at ??:0
@ 0x7fe85e06ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe8607d8104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe8607d5d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe865fc91cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe865fc948e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe865fc9594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe865f8fc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe865f99ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe85ff390d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe85ff2e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe860d926db start_thread at ??:0
@ 0x7fe85cb6961f clone at ??:0
W20260620 19:11:25.482132 17265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:25.482359 17265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:25.482388 17265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:25.483860 17265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:25.483924 17265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:25.483954 17265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:11:25.485536 17265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:39807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=38671
--tserver_master_addrs=127.0.203.62:46105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:25.485823 17265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:25.486099 17265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:25.491416 17304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:25.491434 17305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:25.491427 17307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:25.491695 17265 server_base.cc:1034] running on GCE node
I20260620 19:11:25.491904 17265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:25.492228 17265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:25.493414 17265 hybrid_clock.cc:648] HybridClock initialized: now 1781982685493390 us; error 38 us; skew 500 ppm
Jun 20 19:11:25 dist-test-slave-wd30 krb5kdc[16126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:25.496308 17265 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:11:25.496943 17265 webserver.cc:460] Webserver started at http://127.0.203.1:38671/ using document root <none> and password file <none>
I20260620 19:11:25.497190 17265 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:25.497262 17265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:25.498728 17265 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:11:25.499583 17314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:25.499799 17265 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 19:11:25.499910 17265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "1e931308008d429cab5bc8a64877424f"
format_stamp: "Formatted at 2026-06-20 19:10:43 on dist-test-slave-wd30"
server_key: "00dfdc59af68ef5c9eb22746ffd4cccd"
server_key_iv: "ba4ba3634e53a0a0dbd1cb72d1f0175c"
server_key_version: "encryptionkey@0"
I20260620 19:11:25.500305 17265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:25.523952 17265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:25.526527 17265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:25.526723 17265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:25.527016 17265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:25.527571 17321 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:11:25.529640 17265 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:11:25.529734 17265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 19:11:25.529785 17265 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:11:25.530627 17265 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:11:25.530779 17321 tablet_bootstrap.cc:492] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f: Bootstrap starting.
I20260620 19:11:25.530799 17265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 19:11:25.538146 17265 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:39807
I20260620 19:11:25.538575 17265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:11:25.541939 17321 log.cc:824] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:25.542248 17428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:39807 every 8 connection(s)
I20260620 19:11:25.544911 17321 tablet_bootstrap.cc:492] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:11:25.545310 17321 tablet_bootstrap.cc:492] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f: Bootstrap complete.
I20260620 19:11:25.546602 17321 ts_tablet_manager.cc:1397] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.005s
I20260620 19:11:25.546898 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 17265
I20260620 19:11:25.547199 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:44409
--local_ip_for_outbound_sockets=127.0.203.2
--tserver_master_addrs=127.0.203.62:46105
--webserver_port=40031
--webserver_interface=127.0.203.2
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:11:25.548168 17321 raft_consensus.cc:348] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } } peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } }
I20260620 19:11:25.548522 17321 raft_consensus.cc:729] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e931308008d429cab5bc8a64877424f, State: Initialized, Role: FOLLOWER
I20260620 19:11:25.548687 17321 consensus_queue.cc:260] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } } peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } }
I20260620 19:11:25.548959 17321 ts_tablet_manager.cc:1428] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f: Time spent starting tablet: real 0.002s user 0.003s sys 0.003s
W20260620 19:11:25.618917 17431 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:11:25.539357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.1:49513)
0620 19:11:25.539563 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:25.539606 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:25.540057 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:25.618702 (+ 78645us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":123,"threads_started":1}
W20260620 19:11:25.619220 17429 heartbeater.cc:644] Failed to heartbeat to 127.0.203.62:46105 (0 consecutive failures): Network error: Failed to ping master at 127.0.203.62:46105: Client connection negotiation failed: client connection to 127.0.203.62:46105: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:11:25.619660 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:25.619514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.1:39705)
0620 19:11:25.619578 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:25.619631 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 19:11:25.619969 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:25.619897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.1:47701)
0620 19:11:25.619925 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:25.619950 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260620 19:11:25.620088 17429 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:11:25.647198 17433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:25.647421 17433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:25.647446 17433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:25.648649 17433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:25.648698 17433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:25.648725 17433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:11:25.650151 17433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:44409
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=40031
--tserver_master_addrs=127.0.203.62:46105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:25.650394 17433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:25.650624 17433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:25.655671 17442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:25.655699 17440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:25.655678 17439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:25.655987 17433 server_base.cc:1034] running on GCE node
I20260620 19:11:25.656255 17433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:25.656481 17433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:25.657658 17433 hybrid_clock.cc:648] HybridClock initialized: now 1781982685657635 us; error 37 us; skew 500 ppm
Jun 20 19:11:25 dist-test-slave-wd30 krb5kdc[16126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:25.660185 17433 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:11:25.660645 17433 webserver.cc:460] Webserver started at http://127.0.203.2:40031/ using document root <none> and password file <none>
I20260620 19:11:25.660881 17433 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:25.660952 17433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:25.662230 17433 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:11:25.662873 17449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:25.663023 17433 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:11:25.663105 17433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0"
format_stamp: "Formatted at 2026-06-20 19:10:43 on dist-test-slave-wd30"
server_key: "62684ef19fc6df101406cbde57e82130"
server_key_iv: "779c3f08b3e82327c91ffdf55e4f0a85"
server_key_version: "encryptionkey@0"
I20260620 19:11:25.663479 17433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:25.679903 17433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:25.682456 17433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:25.682582 17433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:25.682792 17433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:25.683224 17433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:25.683269 17433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:25.683293 17433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:25.683338 17433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:25.689337 17433 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:44409
I20260620 19:11:25.689383 17562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:44409 every 8 connection(s)
I20260620 19:11:25.689736 17433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:11:25.690922 17565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:25.690409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.2:44623)
0620 19:11:25.690640 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:25.690744 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":70,"threads_started":1}
W20260620 19:11:25.691391 17563 heartbeater.cc:644] Failed to heartbeat to 127.0.203.62:46105 (0 consecutive failures): Network error: Failed to ping master at 127.0.203.62:46105: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
I20260620 19:11:25.691707 17565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:25.691610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.2:42349)
0620 19:11:25.691661 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:25.691688 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 19:11:25.692075 17565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:25.691999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.2:53985)
0620 19:11:25.692039 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:25.692060 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 19:11:25.692198 17563 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:11:25.693329 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 17433
I20260620 19:11:25.693631 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:43299
--local_ip_for_outbound_sockets=127.0.203.3
--tserver_master_addrs=127.0.203.62:46105
--webserver_port=43667
--webserver_interface=127.0.203.3
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:25.774811 17566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:25.775041 17566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:25.775068 17566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:25.776332 17566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:25.776387 17566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:25.776417 17566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:11:25.777920 17566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:43299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=43667
--tserver_master_addrs=127.0.203.62:46105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:25.778143 17566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:25.778350 17566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:25.783878 17571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:25.783883 17572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:25.783885 17574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:25.784201 17566 server_base.cc:1034] running on GCE node
I20260620 19:11:25.784420 17566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:25.784714 17566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:25.785913 17566 hybrid_clock.cc:648] HybridClock initialized: now 1781982685785905 us; error 45 us; skew 500 ppm
Jun 20 19:11:25 dist-test-slave-wd30 krb5kdc[16126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:25.788568 17566 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:11:25.789094 17566 webserver.cc:460] Webserver started at http://127.0.203.3:43667/ using document root <none> and password file <none>
I20260620 19:11:25.789354 17566 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:25.789427 17566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:25.790881 17566 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:11:25.791630 17581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:25.791862 17566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 19:11:25.791975 17566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "d6785bfe80704762a95cc9bf281dd210"
format_stamp: "Formatted at 2026-06-20 19:10:44 on dist-test-slave-wd30"
server_key: "136c180a87920cc264ae1ea137fd76bd"
server_key_iv: "7971f97415af26e7935a35e03698bb74"
server_key_version: "encryptionkey@0"
I20260620 19:11:25.792374 17566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:25.810668 17566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:25.813122 17566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:25.813272 17566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:25.813468 17566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:25.813849 17566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:25.813922 17566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:25.813967 17566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:25.813994 17566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:25.820176 17566 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:43299
I20260620 19:11:25.820225 17694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:43299 every 8 connection(s)
I20260620 19:11:25.820628 17566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:11:25.821815 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:25.821425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.3:39179)
0620 19:11:25.821590 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:25.821666 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
W20260620 19:11:25.822288 17695 heartbeater.cc:644] Failed to heartbeat to 127.0.203.62:46105 (0 consecutive failures): Network error: Failed to ping master at 127.0.203.62:46105: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
I20260620 19:11:25.822654 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:25.822530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.3:55267)
0620 19:11:25.822601 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:25.822631 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 19:11:25.822976 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:25.822890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.3:58761)
0620 19:11:25.822935 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:25.822958 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 19:11:25.823108 17695 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:11:25.829274 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 17566
I20260620 19:11:26.620771 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:26.620492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.1:47073)
0620 19:11:26.620648 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:26.620699 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":72,"threads_started":1}
I20260620 19:11:26.692862 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:26.692605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.2:36695)
0620 19:11:26.692755 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:26.692799 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
I20260620 19:11:26.823747 17701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:26.823518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.3:56969)
0620 19:11:26.823654 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:26.823691 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260620 19:11:26.984992 17702 raft_consensus.cc:482] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:11:26.985177 17702 raft_consensus.cc:504] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } } peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } }
I20260620 19:11:26.985617 17702 leader_election.cc:288] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f10fd16e9d0e4285b4c8de19cb7af0e0 (127.0.203.2:44409), d6785bfe80704762a95cc9bf281dd210 (127.0.203.3:43299)
Jun 20 19:11:26 dist-test-slave-wd30 krb5kdc[16126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.3@KRBTEST.COM
Jun 20 19:11:26 dist-test-slave-wd30 krb5kdc[16126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.2@KRBTEST.COM
I20260620 19:11:26.997471 17703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:26.985906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:42211 (local address 127.0.203.2:44409)
0620 19:11:26.986089 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:26.986101 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:26.986142 (+ 41us) server_negotiation.cc:407] Connection header received
0620 19:11:26.986676 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:26.986678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:26.986723 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:26.986883 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:26.987740 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:26.988659 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:26.989746 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:26.990175 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:26.990316 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:26.993496 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:26.993500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:26.993500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:26.993511 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:26.995690 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:26.996135 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:26.996139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:26.996140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:26.996243 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:26.996539 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:26.996540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:26.996541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:26.996808 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:26.996908 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:26.997211 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 19:11:26.997342 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
I20260620 19:11:26.997460 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:26.985810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:44409 (local address 127.0.203.1:42211)
0620 19:11:26.985865 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:26.985879 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:26.985949 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:26.986935 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:26.986938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:26.987402 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:26.987602 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:26.987609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:26.988808 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:26.988810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:26.989634 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:26.989637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:26.990293 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:26.990294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:26.990445 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:26.991114 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:11:26.991170 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:26.993353 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:26.995760 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:26.995761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:26.995761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:26.996052 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:26.996326 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:26.996327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:26.996327 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:26.996457 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:26.996886 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:26.996894 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:26.997042 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 19:11:26.997152 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 19:11:26.997355 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":56}
I20260620 19:11:26.997511 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:26.986046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:58213 (local address 127.0.203.3:43299)
0620 19:11:26.986233 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:26.986243 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:26.986282 (+ 39us) server_negotiation.cc:407] Connection header received
0620 19:11:26.986751 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:26.986753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:26.986801 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:26.986964 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:26.987745 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:26.988659 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:26.989762 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:26.990202 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:26.990347 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:26.993081 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:26.993085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:26.993086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:26.993099 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:26.995339 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:26.996182 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:26.996185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:26.996185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:26.996315 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:26.996582 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:26.996583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:26.996584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:26.996875 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:26.996975 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:26.997222 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 19:11:26.997384 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":68,"threads_started":1}
I20260620 19:11:26.997586 17704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:26.985820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:43299 (local address 127.0.203.1:58213)
0620 19:11:26.986188 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:26.986199 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:26.986257 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:26.986963 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:26.986964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:26.987408 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:26.987602 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:26.987609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:26.988820 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:26.988822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:26.989635 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:26.989637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:26.990317 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:26.990318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:26.990412 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:26.991198 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:11:26.991203 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:26.992921 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:26.995590 (+ 2669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:26.995599 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:26.995604 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:26.996083 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:26.996386 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:26.996387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:26.996387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:26.996466 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:26.996956 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:26.996957 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:26.997052 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 19:11:26.997151 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 19:11:26.997441 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":368,"thread_start_us":47,"threads_started":1}
W20260620 19:11:26.998413 17316 leader_election.cc:341] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6785bfe80704762a95cc9bf281dd210 (127.0.203.3:43299): Not found: Tablet not found: bdd43259792b40b3a32df8437cb466c3
W20260620 19:11:26.998975 17317 leader_election.cc:341] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f10fd16e9d0e4285b4c8de19cb7af0e0 (127.0.203.2:44409): Not found: Tablet not found: bdd43259792b40b3a32df8437cb466c3
I20260620 19:11:26.999019 17317 leader_election.cc:302] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e931308008d429cab5bc8a64877424f; no voters: d6785bfe80704762a95cc9bf281dd210, f10fd16e9d0e4285b4c8de19cb7af0e0
I20260620 19:11:26.999182 17702 raft_consensus.cc:2726] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:11:27.621634 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:27.621353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.1:54249)
0620 19:11:27.621535 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:27.621586 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":73,"threads_started":1}
I20260620 19:11:27.693607 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:27.693362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.2:49743)
0620 19:11:27.693515 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:27.693560 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
I20260620 19:11:27.824579 17708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:27.824371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.3:58237)
0620 19:11:27.824493 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:27.824531 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260620 19:11:28.622464 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:28.622222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.1:52907)
0620 19:11:28.622372 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:28.622416 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":64,"threads_started":1}
I20260620 19:11:28.694486 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:28.694210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.2:34161)
0620 19:11:28.694365 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:28.694403 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260620 19:11:28.804097 17711 raft_consensus.cc:482] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:11:28.804260 17711 raft_consensus.cc:504] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } } peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } }
I20260620 19:11:28.804448 17711 leader_election.cc:288] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f10fd16e9d0e4285b4c8de19cb7af0e0 (127.0.203.2:44409), d6785bfe80704762a95cc9bf281dd210 (127.0.203.3:43299)
W20260620 19:11:28.804881 17316 leader_election.cc:341] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6785bfe80704762a95cc9bf281dd210 (127.0.203.3:43299): Not found: Tablet not found: bdd43259792b40b3a32df8437cb466c3
W20260620 19:11:28.804970 17317 leader_election.cc:341] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f10fd16e9d0e4285b4c8de19cb7af0e0 (127.0.203.2:44409): Not found: Tablet not found: bdd43259792b40b3a32df8437cb466c3
I20260620 19:11:28.805040 17316 leader_election.cc:302] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e931308008d429cab5bc8a64877424f; no voters: d6785bfe80704762a95cc9bf281dd210, f10fd16e9d0e4285b4c8de19cb7af0e0
I20260620 19:11:28.805156 17711 raft_consensus.cc:2726] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:11:28.825331 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:28.825103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.3:53599)
0620 19:11:28.825248 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:28.825282 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260620 19:11:29.623404 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:29.623074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.1:36575)
0620 19:11:29.623300 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:29.623347 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":149,"threads_started":1}
I20260620 19:11:29.695356 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:29.695098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.2:47549)
0620 19:11:29.695263 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:29.695308 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":100,"threads_started":1}
I20260620 19:11:29.826156 17715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:29.825916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.3:35419)
0620 19:11:29.826060 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:29.826107 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260620 19:11:30.624439 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:30.624182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.1:33027)
0620 19:11:30.624342 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:30.624377 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
I20260620 19:11:30.696353 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:30.696060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.2:33743)
0620 19:11:30.696262 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:30.696305 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":137,"threads_started":1}
I20260620 19:11:30.826913 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:30.826667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.3:57679)
0620 19:11:30.826815 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:30.826863 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260620 19:11:31.625168 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:31.624933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.1:32801)
0620 19:11:31.625077 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:31.625119 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260620 19:11:31.697039 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:31.696833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.2:33221)
0620 19:11:31.696953 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:31.696991 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":87,"threads_started":1}
I20260620 19:11:31.827661 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:31.827421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.3:49461)
0620 19:11:31.827569 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:31.827612 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260620 19:11:32.025681 17722 raft_consensus.cc:482] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:11:32.025856 17722 raft_consensus.cc:504] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } } peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } }
I20260620 19:11:32.026088 17722 leader_election.cc:288] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f10fd16e9d0e4285b4c8de19cb7af0e0 (127.0.203.2:44409), d6785bfe80704762a95cc9bf281dd210 (127.0.203.3:43299)
W20260620 19:11:32.026520 17316 leader_election.cc:341] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6785bfe80704762a95cc9bf281dd210 (127.0.203.3:43299): Not found: Tablet not found: bdd43259792b40b3a32df8437cb466c3
W20260620 19:11:32.026608 17317 leader_election.cc:341] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f10fd16e9d0e4285b4c8de19cb7af0e0 (127.0.203.2:44409): Not found: Tablet not found: bdd43259792b40b3a32df8437cb466c3
I20260620 19:11:32.026692 17317 leader_election.cc:302] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e931308008d429cab5bc8a64877424f; no voters: d6785bfe80704762a95cc9bf281dd210, f10fd16e9d0e4285b4c8de19cb7af0e0
I20260620 19:11:32.026763 17722 raft_consensus.cc:2726] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:11:32.626037 17723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:32.625743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.1:58169)
0620 19:11:32.625933 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:32.625981 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":77,"threads_started":1}
I20260620 19:11:32.697679 17724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:32.697478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.2:34889)
0620 19:11:32.697598 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:32.697634 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260620 19:11:32.828521 17725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:32.828253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.3:40819)
0620 19:11:32.828418 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:32.828467 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":66,"threads_started":1}
I20260620 19:11:33.626878 17726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:33.626650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.1:51435)
0620 19:11:33.626784 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:33.626829 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260620 19:11:33.698355 17727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:33.698126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.2:40965)
0620 19:11:33.698267 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:33.698310 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
I20260620 19:11:33.829346 17728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:33.829102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.3:46671)
0620 19:11:33.829247 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:33.829299 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260620 19:11:34.627631 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:34.627407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.1:53743)
0620 19:11:34.627549 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:34.627585 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260620 19:11:34.699227 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:34.698883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.2:51927)
0620 19:11:34.699065 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:34.699112 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":98,"threads_started":1}
I20260620 19:11:34.830107 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:34.829877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.3:47281)
0620 19:11:34.830022 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:34.830059 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260620 19:11:35.628384 17732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:35.628150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.1:57443)
0620 19:11:35.628294 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:35.628337 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260620 19:11:35.699901 17733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:35.699666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.2:51615)
0620 19:11:35.699817 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:35.699854 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260620 19:11:35.830910 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:35.830655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.3:50781)
0620 19:11:35.830818 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:35.830863 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
I20260620 19:11:36.499681 17735 raft_consensus.cc:482] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:11:36.499809 17735 raft_consensus.cc:504] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10fd16e9d0e4285b4c8de19cb7af0e0" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44409 } } peers { permanent_uuid: "d6785bfe80704762a95cc9bf281dd210" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43299 } } peers { permanent_uuid: "1e931308008d429cab5bc8a64877424f" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 39807 } }
I20260620 19:11:36.500010 17735 leader_election.cc:288] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f10fd16e9d0e4285b4c8de19cb7af0e0 (127.0.203.2:44409), d6785bfe80704762a95cc9bf281dd210 (127.0.203.3:43299)
W20260620 19:11:36.500427 17317 leader_election.cc:341] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f10fd16e9d0e4285b4c8de19cb7af0e0 (127.0.203.2:44409): Not found: Tablet not found: bdd43259792b40b3a32df8437cb466c3
W20260620 19:11:36.500566 17316 leader_election.cc:341] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6785bfe80704762a95cc9bf281dd210 (127.0.203.3:43299): Not found: Tablet not found: bdd43259792b40b3a32df8437cb466c3
I20260620 19:11:36.500695 17316 leader_election.cc:302] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e931308008d429cab5bc8a64877424f; no voters: d6785bfe80704762a95cc9bf281dd210, f10fd16e9d0e4285b4c8de19cb7af0e0
I20260620 19:11:36.500790 17735 raft_consensus.cc:2726] T bdd43259792b40b3a32df8437cb466c3 P 1e931308008d429cab5bc8a64877424f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:11:36.629190 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:36.628965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.1:47163)
0620 19:11:36.629096 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:36.629142 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260620 19:11:36.700621 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:36.700382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.2:48529)
0620 19:11:36.700533 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:36.700575 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":105,"threads_started":1}
I20260620 19:11:36.831733 17738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:36.831460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.3:35725)
0620 19:11:36.831618 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:36.831667 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260620 19:11:37.629916 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:37.629704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.1:35785)
0620 19:11:37.629832 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:37.629865 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260620 19:11:37.701454 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:37.701177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.2:57587)
0620 19:11:37.701327 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:37.701373 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260620 19:11:37.832490 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:37.832277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.3:52529)
0620 19:11:37.832406 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:37.832443 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":59,"threads_started":1}
I20260620 19:11:38.630597 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:38.630352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.1:40401)
0620 19:11:38.630511 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:38.630550 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":60,"threads_started":1}
I20260620 19:11:38.702243 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:38.702012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.2:46233)
0620 19:11:38.702147 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:38.702194 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260620 19:11:38.833245 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:38.832995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.3:39837)
0620 19:11:38.833147 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:38.833198 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
I20260620 19:11:39.631485 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:39.631247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.1:58851)
0620 19:11:39.631394 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:39.631435 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
I20260620 19:11:39.702996 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:39.702774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.2:32971)
0620 19:11:39.702903 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:39.702950 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260620 19:11:39.833963 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:39.833749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.3:35077)
0620 19:11:39.833876 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:39.833915 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260620 19:11:40.632336 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.632077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.1:57743)
0620 19:11:40.632244 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:40.632287 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":79,"threads_started":1}
I20260620 19:11:40.703763 17749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.703561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.2:40409)
0620 19:11:40.703672 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:40.703708 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
I20260620 19:11:40.834738 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.834531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:46105 (local address 127.0.203.3:41177)
0620 19:11:40.834656 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:40.834691 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.62:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260620 19:11:40.837599 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 17265
I20260620 19:11:40.842403 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 17433
I20260620 19:11:40.846571 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 17566
2026-06-20T19:11:40Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58460 ms)
[----------] 30 tests from SecurityITest (232316 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:11:40 dist-test-slave-wd30 krb5kdc[17753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:11:40 dist-test-slave-wd30 krb5kdc[17753](info): set up 2 sockets
Jun 20 19:11:40 dist-test-slave-wd30 krb5kdc[17753](info): commencing operation
krb5kdc: starting...
W20260620 19:11:41.890887 812 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:11:41 dist-test-slave-wd30 krb5kdc[17753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:11:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:11:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:11:42.008285 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:36139
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:36139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:42.090197 17766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:42.090436 17766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:42.090483 17766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:42.091723 17766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:11:42.091786 17766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:42.091810 17766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:42.091838 17766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:11:42.091871 17766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:11:42.093454 17766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:36139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:36139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:42.093719 17766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:42.093945 17766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:42.099473 17772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:42.099525 17771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:42.099663 17774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:42.100246 17766 server_base.cc:1034] running on GCE node
I20260620 19:11:42.100466 17766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:42.100745 17766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:42.101912 17766 hybrid_clock.cc:648] HybridClock initialized: now 1781982702101896 us; error 28 us; skew 500 ppm
Jun 20 19:11:42 dist-test-slave-wd30 krb5kdc[17753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:42.104651 17766 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:11:42.105023 17766 webserver.cc:460] Webserver started at http://127.0.203.62:45961/ using document root <none> and password file <none>
I20260620 19:11:42.105207 17766 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:42.105247 17766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:42.105333 17766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:42.106339 17766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "49287045e22d4dceb6262782767570f1"
format_stamp: "Formatted at 2026-06-20 19:11:42 on dist-test-slave-wd30"
server_key: "425e3fc0d63b953272256a58ad2b5393"
server_key_iv: "5ae4ffb233f4797d15ff49d102dbb030"
server_key_version: "encryptionkey@0"
I20260620 19:11:42.106632 17766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "49287045e22d4dceb6262782767570f1"
format_stamp: "Formatted at 2026-06-20 19:11:42 on dist-test-slave-wd30"
server_key: "425e3fc0d63b953272256a58ad2b5393"
server_key_iv: "5ae4ffb233f4797d15ff49d102dbb030"
server_key_version: "encryptionkey@0"
I20260620 19:11:42.107959 17766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:11:42.108613 17781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:42.108807 17766 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:11:42.108899 17766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "49287045e22d4dceb6262782767570f1"
format_stamp: "Formatted at 2026-06-20 19:11:42 on dist-test-slave-wd30"
server_key: "425e3fc0d63b953272256a58ad2b5393"
server_key_iv: "5ae4ffb233f4797d15ff49d102dbb030"
server_key_version: "encryptionkey@0"
I20260620 19:11:42.108971 17766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:42.142081 17766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:42.144235 17766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:42.144383 17766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:42.148424 17766 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:36139
I20260620 19:11:42.148489 17833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:36139 every 8 connection(s)
I20260620 19:11:42.148836 17766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:11:42.149708 17834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:42.152045 17834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1: Bootstrap starting.
I20260620 19:11:42.152693 17834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:42.153002 17834 log.cc:824] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:42.153012 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 17766
I20260620 19:11:42.153146 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:11:42.153342 812 external_mini_cluster.cc:1428] Setting key 687415eafc11bf18580f4072870179b9
I20260620 19:11:42.153806 17834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1: No bootstrap required, opened a new log
I20260620 19:11:42.155613 17834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49287045e22d4dceb6262782767570f1" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36139 } }
I20260620 19:11:42.155781 17834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:42.155818 17834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49287045e22d4dceb6262782767570f1, State: Initialized, Role: FOLLOWER
I20260620 19:11:42.155982 17834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49287045e22d4dceb6262782767570f1" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36139 } }
I20260620 19:11:42.156075 17834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:11:42.156111 17834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:11:42.156162 17834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:42.157111 17834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49287045e22d4dceb6262782767570f1" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36139 } }
I20260620 19:11:42.157200 17834 leader_election.cc:302] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49287045e22d4dceb6262782767570f1; no voters:
I20260620 19:11:42.157406 17834 leader_election.cc:288] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:11:42.157429 17839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:42.157558 17839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1 [term 1 LEADER]: Becoming Leader. State: Replica: 49287045e22d4dceb6262782767570f1, State: Running, Role: LEADER
I20260620 19:11:42.157701 17834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:11:42.157655 17839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49287045e22d4dceb6262782767570f1" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36139 } }
I20260620 19:11:42.158013 17839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49287045e22d4dceb6262782767570f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49287045e22d4dceb6262782767570f1" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36139 } } }
I20260620 19:11:42.158075 17839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1 [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:42.158308 17841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49287045e22d4dceb6262782767570f1. Latest consensus state: current_term: 1 leader_uuid: "49287045e22d4dceb6262782767570f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49287045e22d4dceb6262782767570f1" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36139 } } }
I20260620 19:11:42.158368 17841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1 [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:42.158527 17843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:11:42.159039 17843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 19:11:42 dist-test-slave-wd30 krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:11:42.165570 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:42.153860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.0.203.62:36139)
0620 19:11:42.154017 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:42.154025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:42.154050 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:11:42.155599 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:42.155606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:42.155658 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:42.155790 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:42.156610 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.157736 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.158847 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.159632 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.159742 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:42.162362 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:42.162366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:42.162366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:42.162377 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:42.164035 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:42.164402 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:42.164405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:42.164406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:42.164519 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:42.164733 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:42.164735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:42.164735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:42.164982 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:42.165154 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:42.165260 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 19:11:42.165407 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
I20260620 19:11:42.166153 17843 catalog_manager.cc:1347] Generated new cluster ID: db7172361253451080527b10d38b55bb
I20260620 19:11:42.166206 17843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:11:42.199916 17843 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:11:42.200687 17843 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:11:42.204911 17843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1: Generated new TSK 0
I20260620 19:11:42.205125 17843 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
I20260620 19:11:42.240283 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36139
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:42.320811 17860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:42.321027 17860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:42.321050 17860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:42.322232 17860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:42.322273 17860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:42.322300 17860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:11:42.323817 17860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:42.324069 17860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:42.324306 17860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:42.329244 17868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:42.329244 17865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:42.329242 17866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:42.329340 17860 server_base.cc:1034] running on GCE node
I20260620 19:11:42.329762 17860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:42.330008 17860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:42.331175 17860 hybrid_clock.cc:648] HybridClock initialized: now 1781982702331166 us; error 32 us; skew 500 ppm
Jun 20 19:11:42 dist-test-slave-wd30 krb5kdc[17753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:42.333638 17860 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:11:42.334101 17860 webserver.cc:460] Webserver started at http://127.0.203.1:45999/ using document root <none> and password file <none>
I20260620 19:11:42.334343 17860 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:42.334409 17860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:42.334554 17860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:42.335672 17860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "28331117969140e69821ffa35d57fde3"
format_stamp: "Formatted at 2026-06-20 19:11:42 on dist-test-slave-wd30"
server_key: "7ae0f4c6fcba366d9afa75c9b1d98c48"
server_key_iv: "caf21ac8553d7a925ae54c402787d8f2"
server_key_version: "encryptionkey@0"
I20260620 19:11:42.336028 17860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "28331117969140e69821ffa35d57fde3"
format_stamp: "Formatted at 2026-06-20 19:11:42 on dist-test-slave-wd30"
server_key: "7ae0f4c6fcba366d9afa75c9b1d98c48"
server_key_iv: "caf21ac8553d7a925ae54c402787d8f2"
server_key_version: "encryptionkey@0"
I20260620 19:11:42.337332 17860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:11:42.338032 17875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:42.338187 17860 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:11:42.338289 17860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "28331117969140e69821ffa35d57fde3"
format_stamp: "Formatted at 2026-06-20 19:11:42 on dist-test-slave-wd30"
server_key: "7ae0f4c6fcba366d9afa75c9b1d98c48"
server_key_iv: "caf21ac8553d7a925ae54c402787d8f2"
server_key_version: "encryptionkey@0"
I20260620 19:11:42.338361 17860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:42.345391 17860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:42.347304 17860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:42.347450 17860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:42.347666 17860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:42.347975 17860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:42.348035 17860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:42.348078 17860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:42.348104 17860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:42.353797 17860 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:40137
I20260620 19:11:42.353849 17988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:40137 every 8 connection(s)
I20260620 19:11:42.354158 17860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:11:42.355016 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 17860
I20260620 19:11:42.355145 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:11:42.355383 812 external_mini_cluster.cc:1428] Setting key 50cadeecd6901c47b0d05fe39bf3a662
Jun 20 19:11:42 dist-test-slave-wd30 krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.203.2@KRBTEST.COM; defaulting to no policy
I20260620 19:11:42.365252 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:42.354915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:39503 (local address 127.0.203.62:36139)
0620 19:11:42.354990 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:42.354992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:42.355222 (+ 230us) server_negotiation.cc:407] Connection header received
0620 19:11:42.356436 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:42.356437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:42.356449 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:42.356491 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:42.357374 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.357879 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.359411 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.359785 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.359854 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:42.362258 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:42.362259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:42.362260 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:42.362270 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:42.363765 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:42.364201 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:42.364202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:42.364202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:42.364262 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:42.364543 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:42.364544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:42.364545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:42.364663 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:42.364738 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:42.365021 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 19:11:42.365109 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:11:42.365386 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:42.354876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36139 (local address 127.0.203.1:39503)
0620 19:11:42.355064 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:42.355140 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:42.356326 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:42.356563 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:42.356567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:42.356971 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:42.357281 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:42.357288 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.357979 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.357980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:42.359330 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:42.359334 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.359859 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.359860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:42.359970 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:42.360579 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:11:42.360587 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:42.362149 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:42.363867 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:42.363874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:42.363877 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:42.364105 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:42.364331 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:42.364332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:42.364333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:42.364467 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:42.364729 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:42.364736 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:42.364882 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 19:11:42.364989 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 19:11:42.365157 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":106,"threads_started":1}
I20260620 19:11:42.365893 17989 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36139
I20260620 19:11:42.366003 17989 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:42.366214 17989 heartbeater.cc:511] Master 127.0.203.62:36139 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.2@KRBTEST.COM" created.
I20260620 19:11:42.366842 17798 ts_manager.cc:194] Registered new tserver with Master: 28331117969140e69821ffa35d57fde3 (127.0.203.1:40137)
I20260620 19:11:42.367825 17798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:39503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
I20260620 19:11:42.381657 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36139
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:42.464591 17994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:42.464795 17994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:42.464818 17994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:42.465952 17994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:42.465996 17994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:42.466022 17994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:11:42.467528 17994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:42.467757 17994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:42.468003 17994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:42.472883 18002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:42.472903 18000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:42.472905 17999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:42.473124 17994 server_base.cc:1034] running on GCE node
I20260620 19:11:42.473368 17994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:42.473626 17994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:42.474767 17994 hybrid_clock.cc:648] HybridClock initialized: now 1781982702474757 us; error 30 us; skew 500 ppm
Jun 20 19:11:42 dist-test-slave-wd30 krb5kdc[17753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:42.477394 17994 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:11:42.477862 17994 webserver.cc:460] Webserver started at http://127.0.203.2:33209/ using document root <none> and password file <none>
I20260620 19:11:42.478101 17994 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:42.478169 17994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:42.478322 17994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:42.479468 17994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "8de0ecbd237d438ebf26b0a927ca6f5c"
format_stamp: "Formatted at 2026-06-20 19:11:42 on dist-test-slave-wd30"
server_key: "1dc206a6e71bcc2458c81f2a2a009b07"
server_key_iv: "d756e97f28a0cfbf573ce31bacea0975"
server_key_version: "encryptionkey@0"
I20260620 19:11:42.479823 17994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "8de0ecbd237d438ebf26b0a927ca6f5c"
format_stamp: "Formatted at 2026-06-20 19:11:42 on dist-test-slave-wd30"
server_key: "1dc206a6e71bcc2458c81f2a2a009b07"
server_key_iv: "d756e97f28a0cfbf573ce31bacea0975"
server_key_version: "encryptionkey@0"
I20260620 19:11:42.481146 17994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 19:11:42.481884 18009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:42.482069 17994 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:11:42.482172 17994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "8de0ecbd237d438ebf26b0a927ca6f5c"
format_stamp: "Formatted at 2026-06-20 19:11:42 on dist-test-slave-wd30"
server_key: "1dc206a6e71bcc2458c81f2a2a009b07"
server_key_iv: "d756e97f28a0cfbf573ce31bacea0975"
server_key_version: "encryptionkey@0"
I20260620 19:11:42.482247 17994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:42.489426 17994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:42.491760 17994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:42.491922 17994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:42.492146 17994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:42.492449 17994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:42.492504 17994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:42.492547 17994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:42.492574 17994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:42.498816 17994 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:36201
I20260620 19:11:42.498867 18122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:36201 every 8 connection(s)
I20260620 19:11:42.499516 17994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:11:42.506883 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 17994
I20260620 19:11:42.507011 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:11:42.507289 812 external_mini_cluster.cc:1428] Setting key 37e82c8ccd31e60e72e23500002ab12d
Jun 20 19:11:42 dist-test-slave-wd30 krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:11:42.513445 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:42.500418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:46767 (local address 127.0.203.62:36139)
0620 19:11:42.500514 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:42.500516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:42.502549 (+ 2033us) server_negotiation.cc:407] Connection header received
0620 19:11:42.503968 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:42.503969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:42.503999 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:42.504054 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:42.504755 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.505545 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.507110 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.507451 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.507521 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:42.510279 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:42.510280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:42.510281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:42.510288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:42.511834 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:42.512394 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:42.512395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:42.512395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:42.512445 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:42.512755 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:42.512755 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:42.512756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:42.512867 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:42.512921 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:42.513306 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 19:11:42.513375 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 19:11:42.513803 18125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:42.500367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36139 (local address 127.0.203.2:46767)
0620 19:11:42.502400 (+ 2033us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:42.502448 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:42.503848 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:42.504127 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:42.504131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:42.504445 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:42.504674 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:42.504679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.505639 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.505640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:42.507032 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:42.507035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.507593 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.507594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:42.507750 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:42.508325 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:11:42.508336 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:42.510165 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:42.511933 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:42.511941 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:42.511944 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:42.512316 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:42.512522 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:42.512523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:42.512523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:42.512679 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:42.512955 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:42.512963 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:42.513154 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 19:11:42.513290 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 19:11:42.513501 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1979,"thread_start_us":76,"threads_started":1}
I20260620 19:11:42.514387 18123 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36139
I20260620 19:11:42.514523 18123 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:42.514712 18123 heartbeater.cc:511] Master 127.0.203.62:36139 requested a full tablet report, sending...
I20260620 19:11:42.515184 17798 ts_manager.cc:194] Registered new tserver with Master: 8de0ecbd237d438ebf26b0a927ca6f5c (127.0.203.2:36201)
I20260620 19:11:42.515583 17798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:46767
WARNING: no policy specified for kudu/127.0.203.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
I20260620 19:11:42.534430 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36139
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:42.618074 18128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:42.618327 18128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:42.618366 18128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:42.619592 18128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:42.619663 18128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:42.619707 18128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:11:42.621235 18128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:36139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:42.621517 18128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:42.621747 18128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:42.626888 18133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:42.626900 18134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:42.626905 18136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:42.627187 18128 server_base.cc:1034] running on GCE node
I20260620 19:11:42.627445 18128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:42.627701 18128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:42.628891 18128 hybrid_clock.cc:648] HybridClock initialized: now 1781982702628858 us; error 68 us; skew 500 ppm
Jun 20 19:11:42 dist-test-slave-wd30 krb5kdc[17753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:42.631884 18128 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:11:42.632362 18128 webserver.cc:460] Webserver started at http://127.0.203.3:45717/ using document root <none> and password file <none>
I20260620 19:11:42.632622 18128 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:42.632691 18128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:42.632845 18128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:42.633919 18128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "467fa59fe6394e208169edb641492e21"
format_stamp: "Formatted at 2026-06-20 19:11:42 on dist-test-slave-wd30"
server_key: "8481163a8f0928920566248561e4390b"
server_key_iv: "18d2c97926d2fbc77e3703d7e9286f43"
server_key_version: "encryptionkey@0"
I20260620 19:11:42.634295 18128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "467fa59fe6394e208169edb641492e21"
format_stamp: "Formatted at 2026-06-20 19:11:42 on dist-test-slave-wd30"
server_key: "8481163a8f0928920566248561e4390b"
server_key_iv: "18d2c97926d2fbc77e3703d7e9286f43"
server_key_version: "encryptionkey@0"
I20260620 19:11:42.635804 18128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:11:42.636600 18143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:42.636796 18128 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:11:42.636901 18128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "467fa59fe6394e208169edb641492e21"
format_stamp: "Formatted at 2026-06-20 19:11:42 on dist-test-slave-wd30"
server_key: "8481163a8f0928920566248561e4390b"
server_key_iv: "18d2c97926d2fbc77e3703d7e9286f43"
server_key_version: "encryptionkey@0"
I20260620 19:11:42.636981 18128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:42.653499 18128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:42.655835 18128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:42.656011 18128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:42.656251 18128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:42.656571 18128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:42.656627 18128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:42.656672 18128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:42.656713 18128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:42.662947 18128 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:34221
I20260620 19:11:42.662989 18256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:34221 every 8 connection(s)
I20260620 19:11:42.663364 18128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:11:42.669605 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 18128
Jun 20 19:11:42 dist-test-slave-wd30 krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:11:42.669739 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:11:42.669934 812 external_mini_cluster.cc:1428] Setting key aeab3c10a52302b82f4c0eaf4bce1321
I20260620 19:11:42.673004 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:42.664122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:42741 (local address 127.0.203.62:36139)
0620 19:11:42.664190 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:42.664191 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:42.664376 (+ 185us) server_negotiation.cc:407] Connection header received
0620 19:11:42.665286 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:42.665287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:42.665297 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:42.665357 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:42.666147 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.666696 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.667766 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.667964 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.668020 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:42.670321 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:42.670322 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:42.670323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:42.670331 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:42.671780 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:42.672144 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:42.672145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:42.672145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:42.672193 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:42.672425 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:42.672425 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:42.672426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:42.672526 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:42.672574 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:42.672850 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 19:11:42.672938 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:11:42.673112 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:42.664103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:36139 (local address 127.0.203.3:42741)
0620 19:11:42.664248 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:42.664291 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:42.665193 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:42.665414 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:42.665417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:42.665845 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:42.666054 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:42.666059 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.666778 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.666779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:42.667686 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:42.667688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.668038 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.668039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:42.668135 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:42.668649 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:11:42.668657 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:42.670216 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:42.671856 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:42.671862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:42.671864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:42.672077 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:42.672250 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:42.672250 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:42.672250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:42.672346 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:42.672579 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:42.672584 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:42.672706 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 19:11:42.672789 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 19:11:42.672910 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260620 19:11:42.673629 18257 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36139
I20260620 19:11:42.673717 18257 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:42.673916 18257 heartbeater.cc:511] Master 127.0.203.62:36139 requested a full tablet report, sending...
I20260620 19:11:42.674295 17798 ts_manager.cc:194] Registered new tserver with Master: 467fa59fe6394e208169edb641492e21 (127.0.203.3:34221)
I20260620 19:11:42.674654 17798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:42741
I20260620 19:11:42.681972 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:11:42.689333 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:42.683326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46394 (local address 127.0.203.62:36139)
0620 19:11:42.683382 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:42.683384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:42.683455 (+ 71us) server_negotiation.cc:407] Connection header received
0620 19:11:42.683509 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:42.683510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:42.683518 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:42.683568 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:42.684116 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.684603 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.685399 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.685582 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.685641 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:42.686876 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:42.686878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:42.686878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:42.686886 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:42.688347 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:42.688664 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:42.688665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:42.688665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:42.688717 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:42.688891 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:42.688891 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:42.688891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:42.689044 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:42.689117 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:42.689210 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 19:11:42.689267 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:11:42.690905 17798 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:11:42.692584 17798 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:11:42.699035 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:42.694918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49714 (local address 127.0.203.3:34221)
0620 19:11:42.695159 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:42.695164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:42.695174 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:11:42.695214 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:42.695216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:42.695235 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:42.695317 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:42.695700 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.696851 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.698054 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.698583 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.698668 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:42.698731 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:42.698902 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 19:11:42.698949 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":107,"threads_started":1}
I20260620 19:11:42.699025 18265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:42.694803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:34221 (local address 127.0.0.1:49714)
0620 19:11:42.694985 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:42.695026 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:42.695091 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:42.695325 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:42.695328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:42.695331 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:42.695434 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:42.695501 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.696977 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.696978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:42.697969 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:42.697971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.698678 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.698679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:42.698769 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:42.698784 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 19:11:42.698884 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 19:11:42.698943 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":35,"thread_start_us":72,"threads_started":1}
I20260620 19:11:42.699221 18266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:42.695020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:40137 (local address 127.0.0.1:43674)
0620 19:11:42.695217 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:42.695226 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:42.695267 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:42.695628 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:42.695629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:42.695631 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:42.695682 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:42.695684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.697023 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.697024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:42.698084 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:42.698086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.698974 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.698975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:42.699025 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:42.699028 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:11:42.699075 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 19:11:42.699108 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":47,"threads_started":1}
I20260620 19:11:42.699393 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:42.694804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:36201 (local address 127.0.0.1:38232)
0620 19:11:42.695101 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:42.695151 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:42.695205 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:42.695656 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:42.695656 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:42.695659 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:42.695720 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:42.695722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.696602 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.696603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:42.697792 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:42.697794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.698814 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.698816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:42.698893 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:42.698898 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:11:42.698948 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 19:11:42.699005 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":10,"thread_start_us":73,"threads_started":1}
I20260620 19:11:42.699599 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:42.695246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.0.203.1:40137)
0620 19:11:42.695428 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:42.695432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:42.695443 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:11:42.695494 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:42.695496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:42.695512 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:42.695763 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:42.695845 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.696923 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.698162 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.698880 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.699339 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:42.699421 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:42.699447 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 19:11:42.699499 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":76,"threads_started":1}
I20260620 19:11:42.699765 18191 tablet_service.cc:1467] Processing CreateTablet for tablet c0d93a304cc14a9c863fd981b8f2371a (DEFAULT_TABLE table=test-table [id=f702fffda50a4f14a9c82d20918c76a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:42.700088 18191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d93a304cc14a9c863fd981b8f2371a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:42.700351 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:42.694905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.0.203.2:36201)
0620 19:11:42.695390 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:42.695394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:42.695403 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:11:42.695437 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:42.695447 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:42.695467 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:42.695558 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:42.695801 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.696499 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.697886 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.698650 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.698721 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:42.698777 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:42.699086 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 19:11:42.700250 (+ 1164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":81,"threads_started":1}
I20260620 19:11:42.700384 17923 tablet_service.cc:1467] Processing CreateTablet for tablet c0d93a304cc14a9c863fd981b8f2371a (DEFAULT_TABLE table=test-table [id=f702fffda50a4f14a9c82d20918c76a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:42.700730 17923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d93a304cc14a9c863fd981b8f2371a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:42.700980 18057 tablet_service.cc:1467] Processing CreateTablet for tablet c0d93a304cc14a9c863fd981b8f2371a (DEFAULT_TABLE table=test-table [id=f702fffda50a4f14a9c82d20918c76a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:42.701308 18057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d93a304cc14a9c863fd981b8f2371a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:42.703291 18271 tablet_bootstrap.cc:492] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21: Bootstrap starting.
I20260620 19:11:42.703934 18271 tablet_bootstrap.cc:654] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:42.704506 18271 log.cc:824] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:42.704640 18273 tablet_bootstrap.cc:492] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c: Bootstrap starting.
I20260620 19:11:42.705087 18272 tablet_bootstrap.cc:492] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3: Bootstrap starting.
I20260620 19:11:42.705236 18273 tablet_bootstrap.cc:654] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:42.705298 18271 tablet_bootstrap.cc:492] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21: No bootstrap required, opened a new log
I20260620 19:11:42.705382 18271 ts_tablet_manager.cc:1397] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:11:42.705726 18272 tablet_bootstrap.cc:654] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:42.705927 18273 log.cc:824] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:42.706271 18272 log.cc:824] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:42.706708 18273 tablet_bootstrap.cc:492] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c: No bootstrap required, opened a new log
I20260620 19:11:42.706784 18273 ts_tablet_manager.cc:1397] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:11:42.706995 18272 tablet_bootstrap.cc:492] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3: No bootstrap required, opened a new log
I20260620 19:11:42.707065 18272 ts_tablet_manager.cc:1397] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:11:42.707599 18271 raft_consensus.cc:348] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de0ecbd237d438ebf26b0a927ca6f5c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36201 } } peers { permanent_uuid: "467fa59fe6394e208169edb641492e21" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 34221 } } peers { permanent_uuid: "28331117969140e69821ffa35d57fde3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40137 } }
I20260620 19:11:42.707768 18271 raft_consensus.cc:374] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:42.707809 18271 raft_consensus.cc:729] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 467fa59fe6394e208169edb641492e21, State: Initialized, Role: FOLLOWER
I20260620 19:11:42.707942 18271 consensus_queue.cc:260] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de0ecbd237d438ebf26b0a927ca6f5c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36201 } } peers { permanent_uuid: "467fa59fe6394e208169edb641492e21" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 34221 } } peers { permanent_uuid: "28331117969140e69821ffa35d57fde3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40137 } }
I20260620 19:11:42.708226 18271 ts_tablet_manager.cc:1428] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:11:42.708365 18257 heartbeater.cc:503] Master 127.0.203.62:36139 was elected leader, sending a full tablet report...
I20260620 19:11:42.708868 18273 raft_consensus.cc:348] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de0ecbd237d438ebf26b0a927ca6f5c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36201 } } peers { permanent_uuid: "467fa59fe6394e208169edb641492e21" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 34221 } } peers { permanent_uuid: "28331117969140e69821ffa35d57fde3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40137 } }
I20260620 19:11:42.709055 18273 raft_consensus.cc:374] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:42.709095 18273 raft_consensus.cc:729] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de0ecbd237d438ebf26b0a927ca6f5c, State: Initialized, Role: FOLLOWER
I20260620 19:11:42.709088 18272 raft_consensus.cc:348] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de0ecbd237d438ebf26b0a927ca6f5c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36201 } } peers { permanent_uuid: "467fa59fe6394e208169edb641492e21" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 34221 } } peers { permanent_uuid: "28331117969140e69821ffa35d57fde3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40137 } }
I20260620 19:11:42.709251 18272 raft_consensus.cc:374] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:42.709235 18273 consensus_queue.cc:260] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de0ecbd237d438ebf26b0a927ca6f5c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36201 } } peers { permanent_uuid: "467fa59fe6394e208169edb641492e21" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 34221 } } peers { permanent_uuid: "28331117969140e69821ffa35d57fde3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40137 } }
I20260620 19:11:42.709292 18272 raft_consensus.cc:729] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28331117969140e69821ffa35d57fde3, State: Initialized, Role: FOLLOWER
I20260620 19:11:42.709415 18272 consensus_queue.cc:260] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de0ecbd237d438ebf26b0a927ca6f5c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36201 } } peers { permanent_uuid: "467fa59fe6394e208169edb641492e21" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 34221 } } peers { permanent_uuid: "28331117969140e69821ffa35d57fde3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40137 } }
I20260620 19:11:42.709532 18123 heartbeater.cc:503] Master 127.0.203.62:36139 was elected leader, sending a full tablet report...
I20260620 19:11:42.709676 18272 ts_tablet_manager.cc:1428] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:11:42.709693 17989 heartbeater.cc:503] Master 127.0.203.62:36139 was elected leader, sending a full tablet report...
I20260620 19:11:42.709522 18273 ts_tablet_manager.cc:1428] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260620 19:11:42.750128 18124 tablet.cc:2369] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:11:42.837046 18279 raft_consensus.cc:482] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:11:42.837167 18279 raft_consensus.cc:504] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de0ecbd237d438ebf26b0a927ca6f5c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36201 } } peers { permanent_uuid: "467fa59fe6394e208169edb641492e21" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 34221 } } peers { permanent_uuid: "28331117969140e69821ffa35d57fde3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40137 } }
I20260620 19:11:42.837533 18279 leader_election.cc:288] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8de0ecbd237d438ebf26b0a927ca6f5c (127.0.203.2:36201), 467fa59fe6394e208169edb641492e21 (127.0.203.3:34221)
I20260620 19:11:42.841543 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:42.837807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:43927 (local address 127.0.203.2:36201)
0620 19:11:42.837870 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:42.837873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:42.837886 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:11:42.837921 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:42.837922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:42.837935 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:42.838004 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:42.838189 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.838976 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.840551 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.841166 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.841226 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:42.841235 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:42.841429 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 19:11:42.841474 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:11:42.841543 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:42.837859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:49401 (local address 127.0.203.3:34221)
0620 19:11:42.837904 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:42.837905 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:42.838052 (+ 147us) server_negotiation.cc:407] Connection header received
0620 19:11:42.838133 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:42.838134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:42.838141 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:42.838228 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:42.838363 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.839053 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.840542 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.841147 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.841224 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:42.841234 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:42.841446 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 19:11:42.841481 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:11:42.841543 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:42.837712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:36201 (local address 127.0.203.1:43927)
0620 19:11:42.837791 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:42.837804 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:42.837859 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:42.838027 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:42.838028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:42.838030 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:42.838104 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:42.838106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.839079 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.839080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:42.840452 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:42.840455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.841238 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.841239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:42.841342 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:42.841344 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:11:42.841402 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 19:11:42.841452 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":89}
I20260620 19:11:42.841549 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:42.837755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:34221 (local address 127.0.203.1:49401)
0620 19:11:42.837997 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:42.838005 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:42.838078 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:42.838247 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:42.838248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:42.838249 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:42.838299 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:42.838300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.839151 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.839152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:42.840452 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:42.840455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.841238 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.841239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:42.841335 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:42.841339 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:11:42.841438 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 19:11:42.841467 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":61,"thread_start_us":132,"threads_started":1}
I20260620 19:11:42.841929 18077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d93a304cc14a9c863fd981b8f2371a" candidate_uuid: "28331117969140e69821ffa35d57fde3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de0ecbd237d438ebf26b0a927ca6f5c" is_pre_election: true
I20260620 19:11:42.842165 18077 raft_consensus.cc:2454] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28331117969140e69821ffa35d57fde3 in term 0.
I20260620 19:11:42.842159 18211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d93a304cc14a9c863fd981b8f2371a" candidate_uuid: "28331117969140e69821ffa35d57fde3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "467fa59fe6394e208169edb641492e21" is_pre_election: true
I20260620 19:11:42.842327 18211 raft_consensus.cc:2454] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28331117969140e69821ffa35d57fde3 in term 0.
I20260620 19:11:42.842420 17878 leader_election.cc:302] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28331117969140e69821ffa35d57fde3, 8de0ecbd237d438ebf26b0a927ca6f5c; no voters:
I20260620 19:11:42.842561 18279 raft_consensus.cc:2781] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:11:42.842624 18279 raft_consensus.cc:482] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:11:42.842641 18279 raft_consensus.cc:3037] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:42.843400 18279 raft_consensus.cc:504] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de0ecbd237d438ebf26b0a927ca6f5c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36201 } } peers { permanent_uuid: "467fa59fe6394e208169edb641492e21" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 34221 } } peers { permanent_uuid: "28331117969140e69821ffa35d57fde3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40137 } }
I20260620 19:11:42.843557 18279 leader_election.cc:288] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [CANDIDATE]: Term 1 election: Requested vote from peers 8de0ecbd237d438ebf26b0a927ca6f5c (127.0.203.2:36201), 467fa59fe6394e208169edb641492e21 (127.0.203.3:34221)
I20260620 19:11:42.843736 18211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d93a304cc14a9c863fd981b8f2371a" candidate_uuid: "28331117969140e69821ffa35d57fde3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "467fa59fe6394e208169edb641492e21"
I20260620 19:11:42.843737 18077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d93a304cc14a9c863fd981b8f2371a" candidate_uuid: "28331117969140e69821ffa35d57fde3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de0ecbd237d438ebf26b0a927ca6f5c"
I20260620 19:11:42.843835 18211 raft_consensus.cc:3037] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:42.843832 18077 raft_consensus.cc:3037] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:42.844529 18077 raft_consensus.cc:2454] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28331117969140e69821ffa35d57fde3 in term 1.
I20260620 19:11:42.844534 18211 raft_consensus.cc:2454] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28331117969140e69821ffa35d57fde3 in term 1.
I20260620 19:11:42.844707 17876 leader_election.cc:302] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28331117969140e69821ffa35d57fde3, 467fa59fe6394e208169edb641492e21; no voters:
I20260620 19:11:42.844835 18279 raft_consensus.cc:2781] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:42.844977 18279 raft_consensus.cc:686] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [term 1 LEADER]: Becoming Leader. State: Replica: 28331117969140e69821ffa35d57fde3, State: Running, Role: LEADER
I20260620 19:11:42.845088 18279 consensus_queue.cc:237] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de0ecbd237d438ebf26b0a927ca6f5c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36201 } } peers { permanent_uuid: "467fa59fe6394e208169edb641492e21" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 34221 } } peers { permanent_uuid: "28331117969140e69821ffa35d57fde3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40137 } }
I20260620 19:11:42.845970 17798 catalog_manager.cc:5495] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28331117969140e69821ffa35d57fde3 (127.0.203.1). New cstate: current_term: 1 leader_uuid: "28331117969140e69821ffa35d57fde3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de0ecbd237d438ebf26b0a927ca6f5c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "467fa59fe6394e208169edb641492e21" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28331117969140e69821ffa35d57fde3" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40137 } health_report { overall_health: HEALTHY } } }
W20260620 19:11:42.854911 17990 tablet.cc:2369] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:11:42.880620 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:42.877921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43682 (local address 127.0.203.1:40137)
0620 19:11:42.877986 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:42.877988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:42.877998 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:11:42.878079 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:42.878080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:42.878103 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:42.878189 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:11:42.878387 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.878878 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.879750 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:42.879958 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:42.880035 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:42.880238 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:11:42.880344 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:11:42.880452 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:42.880471 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 19:11:42.880494 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:11:42.882731 18077 raft_consensus.cc:1261] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c [term 1 FOLLOWER]: Refusing update from remote peer 28331117969140e69821ffa35d57fde3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:11:42.882717 18211 raft_consensus.cc:1261] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21 [term 1 FOLLOWER]: Refusing update from remote peer 28331117969140e69821ffa35d57fde3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:11:42.883139 18279 consensus_queue.cc:1035] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "467fa59fe6394e208169edb641492e21" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:11:42.883294 18281 consensus_queue.cc:1035] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0ecbd237d438ebf26b0a927ca6f5c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 36201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:11:42.885761 18284 mvcc.cc:205] Tried to move back new op lower bound from 7299001151004528640 to 7299001150854430720. Current Snapshot: MvccSnapshot[applied={T|T < 7299001151004528640}]
I20260620 19:11:42.886197 18285 mvcc.cc:205] Tried to move back new op lower bound from 7299001151004528640 to 7299001150854430720. Current Snapshot: MvccSnapshot[applied={T|T < 7299001151004528640}]
I20260620 19:11:42.886322 18286 mvcc.cc:205] Tried to move back new op lower bound from 7299001151004528640 to 7299001150854430720. Current Snapshot: MvccSnapshot[applied={T|T < 7299001151004528640}]
I20260620 19:11:42.888517 17797 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:11:42.888616 17797 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:11:42.889505 17797 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49287045e22d4dceb6262782767570f1: Sending DeleteTablet for 3 replicas of tablet c0d93a304cc14a9c863fd981b8f2371a
I20260620 19:11:42.889864 18057 tablet_service.cc:1514] Processing DeleteTablet for tablet c0d93a304cc14a9c863fd981b8f2371a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:42 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:38232
I20260620 19:11:42.889910 17923 tablet_service.cc:1514] Processing DeleteTablet for tablet c0d93a304cc14a9c863fd981b8f2371a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:42 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:43674
I20260620 19:11:42.889964 18191 tablet_service.cc:1514] Processing DeleteTablet for tablet c0d93a304cc14a9c863fd981b8f2371a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:42 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:49714
I20260620 19:11:42.890154 18295 tablet_replica.cc:330] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3: stopping tablet replica
I20260620 19:11:42.890256 18295 raft_consensus.cc:2229] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:11:42.890388 18295 raft_consensus.cc:2258] T c0d93a304cc14a9c863fd981b8f2371a P 28331117969140e69821ffa35d57fde3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:11:42.890488 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 17860
I20260620 19:11:42.890543 18296 tablet_replica.cc:330] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21: stopping tablet replica
I20260620 19:11:42.890640 18296 raft_consensus.cc:2229] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:11:42.890736 18296 raft_consensus.cc:2258] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:11:42.892043 18296 ts_tablet_manager.cc:1905] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:11:42.892282 18294 tablet_replica.cc:330] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c: stopping tablet replica
I20260620 19:11:42.892585 18294 raft_consensus.cc:2229] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:11:42.892851 18294 raft_consensus.cc:2258] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:11:42.893265 18294 ts_tablet_manager.cc:1905] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:11:42.894109 18296 ts_tablet_manager.cc:1918] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:11:42.894187 18296 log.cc:1196] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/c0d93a304cc14a9c863fd981b8f2371a
I20260620 19:11:42.894469 18296 ts_tablet_manager.cc:1939] T c0d93a304cc14a9c863fd981b8f2371a P 467fa59fe6394e208169edb641492e21: Deleting consensus metadata
I20260620 19:11:42.894616 18294 ts_tablet_manager.cc:1918] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:11:42.894701 18294 log.cc:1196] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/c0d93a304cc14a9c863fd981b8f2371a
I20260620 19:11:42.894896 17782 catalog_manager.cc:4841] TS 467fa59fe6394e208169edb641492e21 (127.0.203.3:34221): tablet c0d93a304cc14a9c863fd981b8f2371a (table test-table [id=f702fffda50a4f14a9c82d20918c76a3]) successfully deleted
I20260620 19:11:42.895043 18294 ts_tablet_manager.cc:1939] T c0d93a304cc14a9c863fd981b8f2371a P 8de0ecbd237d438ebf26b0a927ca6f5c: Deleting consensus metadata
I20260620 19:11:42.895483 17784 catalog_manager.cc:4841] TS 8de0ecbd237d438ebf26b0a927ca6f5c (127.0.203.2:36201): tablet c0d93a304cc14a9c863fd981b8f2371a (table test-table [id=f702fffda50a4f14a9c82d20918c76a3]) successfully deleted
W20260620 19:11:42.897481 17782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.1:40137 (error 108)
I20260620 19:11:42.897707 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 17994
I20260620 19:11:42.898605 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:42.897865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:40137 (local address 127.0.0.1:43686)
0620 19:11:42.898537 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:42.898576 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.1:40137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":654}
W20260620 19:11:42.898742 17782 catalog_manager.cc:4584] TS 28331117969140e69821ffa35d57fde3 (127.0.203.1:40137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c0d93a304cc14a9c863fd981b8f2371a: Network error: Client connection negotiation failed: client connection to 127.0.203.1:40137: connect: Connection refused (error 111)
I20260620 19:11:42.902993 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 18128
I20260620 19:11:42.907753 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 17766
2026-06-20T19:11:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2061 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T19:11:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:11:42Z Disabled control of system clock
I20260620 19:11:42.935474 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:45707
--webserver_interface=127.0.203.62
--webserver_port=0
--builtin_ntp_servers=127.0.203.20:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:45707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:11:43.029325 18301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:43.029707 18301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:43.029786 18301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:43.031824 18301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:11:43.031875 18301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:43.031888 18301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:43.031917 18301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:11:43.031970 18301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:11:43.033576 18301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:45707
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:45707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:43.033869 18301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:43.034204 18301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:43.039261 18306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:43.039333 18301 server_base.cc:1034] running on GCE node
W20260620 19:11:43.039431 18309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:43.039249 18307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:43.039877 18301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:43.040258 18301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:43.041445 18301 hybrid_clock.cc:648] HybridClock initialized: now 1781982703041423 us; error 35 us; skew 500 ppm
I20260620 19:11:43.043181 18301 webserver.cc:460] Webserver started at http://127.0.203.62:43401/ using document root <none> and password file <none>
I20260620 19:11:43.043511 18301 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:43.043588 18301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:43.043748 18301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:43.045431 18301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "4a273ce5f79541b4b113eb5b3b79a39d"
format_stamp: "Formatted at 2026-06-20 19:11:43 on dist-test-slave-wd30"
server_key: "a95ad6e7a27577d4a95c4642a2701e71"
server_key_iv: "a08059438704cdce98bfd59950c76cea"
server_key_version: "encryptionkey@0"
I20260620 19:11:43.045912 18301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "4a273ce5f79541b4b113eb5b3b79a39d"
format_stamp: "Formatted at 2026-06-20 19:11:43 on dist-test-slave-wd30"
server_key: "a95ad6e7a27577d4a95c4642a2701e71"
server_key_iv: "a08059438704cdce98bfd59950c76cea"
server_key_version: "encryptionkey@0"
I20260620 19:11:43.047822 18301 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 19:11:43.048561 18315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:43.048804 18301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:11:43.048898 18301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "4a273ce5f79541b4b113eb5b3b79a39d"
format_stamp: "Formatted at 2026-06-20 19:11:43 on dist-test-slave-wd30"
server_key: "a95ad6e7a27577d4a95c4642a2701e71"
server_key_iv: "a08059438704cdce98bfd59950c76cea"
server_key_version: "encryptionkey@0"
I20260620 19:11:43.048997 18301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:43.071820 18301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:43.072171 18301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:43.072325 18301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:43.076323 18301 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:45707
I20260620 19:11:43.076385 18367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:45707 every 8 connection(s)
I20260620 19:11:43.076749 18301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:11:43.077706 18368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:43.080260 18368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d: Bootstrap starting.
I20260620 19:11:43.080350 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 18301
I20260620 19:11:43.080433 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:11:43.080639 812 external_mini_cluster.cc:1428] Setting key 8370fccd885f5dfe83766c68885a345b
I20260620 19:11:43.081025 18368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:43.081437 18368 log.cc:824] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:43.082229 18368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d: No bootstrap required, opened a new log
I20260620 19:11:43.084393 18368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a273ce5f79541b4b113eb5b3b79a39d" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 45707 } }
I20260620 19:11:43.084573 18368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:43.084643 18368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a273ce5f79541b4b113eb5b3b79a39d, State: Initialized, Role: FOLLOWER
I20260620 19:11:43.084784 18368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a273ce5f79541b4b113eb5b3b79a39d" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 45707 } }
I20260620 19:11:43.084856 18368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:11:43.084878 18368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:11:43.084942 18368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:43.085621 18368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a273ce5f79541b4b113eb5b3b79a39d" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 45707 } }
I20260620 19:11:43.085719 18368 leader_election.cc:302] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a273ce5f79541b4b113eb5b3b79a39d; no voters:
I20260620 19:11:43.085904 18368 leader_election.cc:288] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:11:43.085969 18373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:43.086122 18373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d [term 1 LEADER]: Becoming Leader. State: Replica: 4a273ce5f79541b4b113eb5b3b79a39d, State: Running, Role: LEADER
I20260620 19:11:43.086205 18368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:11:43.086253 18373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a273ce5f79541b4b113eb5b3b79a39d" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 45707 } }
I20260620 19:11:43.086897 18375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a273ce5f79541b4b113eb5b3b79a39d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a273ce5f79541b4b113eb5b3b79a39d" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 45707 } } }
I20260620 19:11:43.086889 18373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a273ce5f79541b4b113eb5b3b79a39d. Latest consensus state: current_term: 1 leader_uuid: "4a273ce5f79541b4b113eb5b3b79a39d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a273ce5f79541b4b113eb5b3b79a39d" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 45707 } } }
I20260620 19:11:43.087074 18373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:43.087226 18375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:43.087481 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:43.081182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54382 (local address 127.0.203.62:45707)
0620 19:11:43.081429 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:43.081436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:43.081461 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:11:43.081912 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:43.081919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:43.081980 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:43.082102 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:43.082374 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.083383 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.084550 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.085196 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.085292 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:43.086184 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:43.086188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:43.086188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:43.086404 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:43.086445 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:43.086457 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:43.086525 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:43.086734 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 19:11:43.086816 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":119,"threads_started":1}
I20260620 19:11:43.088112 18382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:11:43.088716 18382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:11:43.090181 18382 catalog_manager.cc:1347] Generated new cluster ID: fc74b25dd71a4e3b92dc377cb2b1eaf6
I20260620 19:11:43.090229 18382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:11:43.103953 18382 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:11:43.104683 18382 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:11:43.108083 18382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d: Generated new TSK 0
I20260620 19:11:43.108260 18382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:11:43.112342 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:45707
--builtin_ntp_servers=127.0.203.20:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:11:43.196368 18392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:43.196586 18392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:43.196609 18392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:43.197813 18392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:43.197855 18392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:43.197881 18392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:11:43.199316 18392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:43.199569 18392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:43.199892 18392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:43.204896 18398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:43.204933 18397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:43.205119 18392 server_base.cc:1034] running on GCE node
W20260620 19:11:43.204902 18400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:43.205425 18392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:43.205698 18392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:43.206844 18392 hybrid_clock.cc:648] HybridClock initialized: now 1781982703206825 us; error 35 us; skew 500 ppm
I20260620 19:11:43.208227 18392 webserver.cc:460] Webserver started at http://127.0.203.1:34999/ using document root <none> and password file <none>
I20260620 19:11:43.208500 18392 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:43.208570 18392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:43.208706 18392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:43.209795 18392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "f3df6b12575447ba9df78816a9c2604a"
format_stamp: "Formatted at 2026-06-20 19:11:43 on dist-test-slave-wd30"
server_key: "ce1ead61d7cb5742b9ff5ab7bccdf017"
server_key_iv: "daf159a3c592b09db7c481dfdf00f870"
server_key_version: "encryptionkey@0"
I20260620 19:11:43.210170 18392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "f3df6b12575447ba9df78816a9c2604a"
format_stamp: "Formatted at 2026-06-20 19:11:43 on dist-test-slave-wd30"
server_key: "ce1ead61d7cb5742b9ff5ab7bccdf017"
server_key_iv: "daf159a3c592b09db7c481dfdf00f870"
server_key_version: "encryptionkey@0"
I20260620 19:11:43.211586 18392 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:11:43.212392 18406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:43.212622 18392 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:11:43.212728 18392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "f3df6b12575447ba9df78816a9c2604a"
format_stamp: "Formatted at 2026-06-20 19:11:43 on dist-test-slave-wd30"
server_key: "ce1ead61d7cb5742b9ff5ab7bccdf017"
server_key_iv: "daf159a3c592b09db7c481dfdf00f870"
server_key_version: "encryptionkey@0"
I20260620 19:11:43.212801 18392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:43.234826 18392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:43.235158 18392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:43.235270 18392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:43.235587 18392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:43.235947 18392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:43.236011 18392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:43.236076 18392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:43.236105 18392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:43.241844 18392 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:40535
I20260620 19:11:43.241923 18519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:40535 every 8 connection(s)
I20260620 19:11:43.242264 18392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:11:43.247700 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 18392
I20260620 19:11:43.247751 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:43.243009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:54069 (local address 127.0.203.62:45707)
0620 19:11:43.243074 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:43.243076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:43.243351 (+ 275us) server_negotiation.cc:407] Connection header received
0620 19:11:43.243910 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:43.243911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:43.243923 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:43.243976 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:43.244674 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.245192 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.246244 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.246451 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.246530 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:43.247449 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:43.247451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:43.247451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:43.247456 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:43.247471 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:43.247475 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:43.247544 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:43.247642 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 19:11:43.247668 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:11:43.247809 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:11:43.247974 18522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:43.242999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:45707 (local address 127.0.203.1:54069)
0620 19:11:43.243221 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:43.243259 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:43.243804 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:43.244056 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:43.244066 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:43.244075 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:43.244575 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:43.244579 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.245299 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.245300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:43.246150 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:43.246153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.246539 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.246541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:43.246675 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:43.247290 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:11:43.247291 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:43.247320 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:11:43.247344 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:43.247549 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:43.247555 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:43.247558 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:11:43.247643 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 19:11:43.247746 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
I20260620 19:11:43.248031 812 external_mini_cluster.cc:1428] Setting key e434874bfde17d6893d5709d96e7da3d
I20260620 19:11:43.248641 18520 heartbeater.cc:347] Connected to a master server at 127.0.203.62:45707
I20260620 19:11:43.248755 18520 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:43.248941 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:45707
--builtin_ntp_servers=127.0.203.20:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:11:43.248963 18520 heartbeater.cc:511] Master 127.0.203.62:45707 requested a full tablet report, sending...
I20260620 19:11:43.249507 18332 ts_manager.cc:194] Registered new tserver with Master: f3df6b12575447ba9df78816a9c2604a (127.0.203.1:40535)
I20260620 19:11:43.250150 18332 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.1:54069
W20260620 19:11:43.331530 18523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:43.331745 18523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:43.331769 18523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:43.332876 18523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:43.332917 18523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:43.332943 18523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:11:43.334342 18523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:43.334564 18523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:43.334827 18523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:43.340008 18528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:43.340046 18529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:43.340037 18531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:43.340205 18523 server_base.cc:1034] running on GCE node
I20260620 19:11:43.340474 18523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:43.340740 18523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:43.341897 18523 hybrid_clock.cc:648] HybridClock initialized: now 1781982703341884 us; error 28 us; skew 500 ppm
I20260620 19:11:43.343289 18523 webserver.cc:460] Webserver started at http://127.0.203.2:39473/ using document root <none> and password file <none>
I20260620 19:11:43.343557 18523 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:43.343603 18523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:43.343775 18523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:43.344884 18523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "b085d0a44ada4537a5eb886e7c26ac0c"
format_stamp: "Formatted at 2026-06-20 19:11:43 on dist-test-slave-wd30"
server_key: "98d48c67eed5cfda43c963b7f25a09c9"
server_key_iv: "2a7e43c6621cf9aa815f54eef8531cb3"
server_key_version: "encryptionkey@0"
I20260620 19:11:43.345182 18523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "b085d0a44ada4537a5eb886e7c26ac0c"
format_stamp: "Formatted at 2026-06-20 19:11:43 on dist-test-slave-wd30"
server_key: "98d48c67eed5cfda43c963b7f25a09c9"
server_key_iv: "2a7e43c6621cf9aa815f54eef8531cb3"
server_key_version: "encryptionkey@0"
I20260620 19:11:43.346513 18523 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:11:43.347361 18537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:43.347544 18523 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:11:43.347640 18523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "b085d0a44ada4537a5eb886e7c26ac0c"
format_stamp: "Formatted at 2026-06-20 19:11:43 on dist-test-slave-wd30"
server_key: "98d48c67eed5cfda43c963b7f25a09c9"
server_key_iv: "2a7e43c6621cf9aa815f54eef8531cb3"
server_key_version: "encryptionkey@0"
I20260620 19:11:43.347713 18523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:43.358134 18523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:43.358480 18523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:43.358633 18523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:43.358884 18523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:43.359349 18523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:43.359392 18523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:43.359416 18523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:43.359462 18523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:43.365378 18523 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:33763
I20260620 19:11:43.365437 18650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:33763 every 8 connection(s)
I20260620 19:11:43.365792 18523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:11:43.371316 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:43.366582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:44851 (local address 127.0.203.62:45707)
0620 19:11:43.366656 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:43.366658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:43.366913 (+ 255us) server_negotiation.cc:407] Connection header received
0620 19:11:43.367483 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:43.367484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:43.367495 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:43.367555 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:43.368208 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.368703 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.369749 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.369935 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.370000 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:43.370986 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:43.370992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:43.370992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:43.370996 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:43.371010 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:43.371014 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:43.371068 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:43.371229 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 19:11:43.371252 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:11:43.371505 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:43.366571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:45707 (local address 127.0.203.2:44851)
0620 19:11:43.366771 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:43.366814 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:43.367382 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:43.367617 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:43.367620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:43.367629 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:43.368132 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:43.368135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.368807 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.368809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:43.369613 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:43.369616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.370027 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.370028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:43.370140 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:43.370828 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:11:43.370828 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:43.370859 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:11:43.370888 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:43.371080 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:43.371084 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:43.371086 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:11:43.371191 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 19:11:43.371281 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
I20260620 19:11:43.372084 18651 heartbeater.cc:347] Connected to a master server at 127.0.203.62:45707
I20260620 19:11:43.372165 18651 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:43.372400 18651 heartbeater.cc:511] Master 127.0.203.62:45707 requested a full tablet report, sending...
I20260620 19:11:43.372820 18332 ts_manager.cc:194] Registered new tserver with Master: b085d0a44ada4537a5eb886e7c26ac0c (127.0.203.2:33763)
I20260620 19:11:43.373242 18332 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.2:44851
I20260620 19:11:43.374146 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 18523
I20260620 19:11:43.374258 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:11:43.374465 812 external_mini_cluster.cc:1428] Setting key b2fea64dc4ffe5f069e3499dd87023e3
I20260620 19:11:43.375358 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:45707
--builtin_ntp_servers=127.0.203.20:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:11:43.457037 18654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:43.457243 18654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:43.457265 18654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:43.458407 18654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:43.458448 18654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:43.458473 18654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:11:43.459877 18654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:43.460116 18654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:43.460433 18654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:43.465698 18659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:43.465658 18662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:43.465732 18654 server_base.cc:1034] running on GCE node
W20260620 19:11:43.465658 18660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:43.466162 18654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:43.466410 18654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:43.467602 18654 hybrid_clock.cc:648] HybridClock initialized: now 1781982703467583 us; error 28 us; skew 500 ppm
I20260620 19:11:43.468976 18654 webserver.cc:460] Webserver started at http://127.0.203.3:33119/ using document root <none> and password file <none>
I20260620 19:11:43.469230 18654 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:43.469326 18654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:43.469482 18654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:43.470558 18654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "ed254c0cd8434b82a07d405b2aec7e7f"
format_stamp: "Formatted at 2026-06-20 19:11:43 on dist-test-slave-wd30"
server_key: "1e771b360792fe096cfa42d259ae8f03"
server_key_iv: "b1a69b28728deb801c7537c7d4839b08"
server_key_version: "encryptionkey@0"
I20260620 19:11:43.470923 18654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "ed254c0cd8434b82a07d405b2aec7e7f"
format_stamp: "Formatted at 2026-06-20 19:11:43 on dist-test-slave-wd30"
server_key: "1e771b360792fe096cfa42d259ae8f03"
server_key_iv: "b1a69b28728deb801c7537c7d4839b08"
server_key_version: "encryptionkey@0"
I20260620 19:11:43.472276 18654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:11:43.472970 18668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:43.473152 18654 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:11:43.473258 18654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "ed254c0cd8434b82a07d405b2aec7e7f"
format_stamp: "Formatted at 2026-06-20 19:11:43 on dist-test-slave-wd30"
server_key: "1e771b360792fe096cfa42d259ae8f03"
server_key_iv: "b1a69b28728deb801c7537c7d4839b08"
server_key_version: "encryptionkey@0"
I20260620 19:11:43.473340 18654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:43.484632 18654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:43.484946 18654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:43.485093 18654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:43.485328 18654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:43.485644 18654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:43.485702 18654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:43.485747 18654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:43.485771 18654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:43.491921 18654 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:43491
I20260620 19:11:43.491963 18781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:43491 every 8 connection(s)
I20260620 19:11:43.492307 18654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:11:43.497581 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:43.493020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:58715 (local address 127.0.203.62:45707)
0620 19:11:43.493083 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:43.493085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:43.493330 (+ 245us) server_negotiation.cc:407] Connection header received
0620 19:11:43.493865 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:43.493866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:43.493877 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:43.493934 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:43.494617 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.495186 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.496201 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.496439 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.496500 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:43.497298 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:43.497299 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:43.497299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:43.497305 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:43.497318 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:43.497322 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:43.497376 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:43.497469 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 19:11:43.497495 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:11:43.497825 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:43.493002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:45707 (local address 127.0.203.3:58715)
0620 19:11:43.493206 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:43.493242 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:43.493760 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:43.494014 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:43.494018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:43.494026 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:43.494522 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:43.494526 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.495315 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.495316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:43.496091 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:43.496094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.496520 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.496521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:43.496649 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:43.497130 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:11:43.497131 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:43.497159 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:11:43.497181 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:43.497379 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:43.497384 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:43.497387 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:11:43.497471 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:11:43.497572 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260620 19:11:43.498291 18782 heartbeater.cc:347] Connected to a master server at 127.0.203.62:45707
I20260620 19:11:43.498394 18782 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:43.498571 18782 heartbeater.cc:511] Master 127.0.203.62:45707 requested a full tablet report, sending...
I20260620 19:11:43.498950 18332 ts_manager.cc:194] Registered new tserver with Master: ed254c0cd8434b82a07d405b2aec7e7f (127.0.203.3:43491)
I20260620 19:11:43.499395 18332 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.3:58715
I20260620 19:11:43.500115 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 18654
I20260620 19:11:43.500186 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:11:43.500334 812 external_mini_cluster.cc:1428] Setting key 345d311c2db8d42346d068f87384a529
I20260620 19:11:43.501145 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:11:43.505244 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:43.502497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54392 (local address 127.0.203.62:45707)
0620 19:11:43.502540 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:43.502542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:43.502655 (+ 113us) server_negotiation.cc:407] Connection header received
0620 19:11:43.502716 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:43.502717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:43.502726 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:43.502767 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:43.502886 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.503424 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.504129 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.504319 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.504367 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:43.505012 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:43.505013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:43.505014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:43.505018 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:43.505028 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:43.505031 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:43.505068 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:43.505121 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 19:11:43.505139 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:11:43.506704 18332 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:11:43.508292 18332 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:11:43.514662 18790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:43.510711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:40535 (local address 127.0.0.1:55406)
0620 19:11:43.511179 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:43.511190 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:43.511234 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:43.511492 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:43.511493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:43.511495 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:43.511524 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:43.511525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.512705 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.512706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:43.513609 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:43.513611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.514332 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.514333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:43.514409 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:43.514421 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 19:11:43.514507 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 19:11:43.514562 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":65,"threads_started":1}
I20260620 19:11:43.514772 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:43.510594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.0.203.3:43491)
0620 19:11:43.510774 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:43.510779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:43.510924 (+ 145us) server_negotiation.cc:407] Connection header received
0620 19:11:43.511025 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:43.511027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:43.511042 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:43.511363 (+ 321us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:43.511426 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.512544 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.513749 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.514435 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.514503 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:43.514574 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:43.514631 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 19:11:43.514678 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260620 19:11:43.514926 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:43.510917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55406 (local address 127.0.203.1:40535)
0620 19:11:43.511104 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:43.511108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:43.511302 (+ 194us) server_negotiation.cc:407] Connection header received
0620 19:11:43.511348 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:43.511350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:43.511366 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:43.511433 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:43.511604 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.512647 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.513700 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.514226 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.514321 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:43.514421 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:43.514805 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 19:11:43.514855 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":78,"threads_started":1}
I20260620 19:11:43.515074 18793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:43.510631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42374 (local address 127.0.203.2:33763)
0620 19:11:43.510850 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:43.510854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:43.511034 (+ 180us) server_negotiation.cc:407] Connection header received
0620 19:11:43.511083 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:43.511085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:43.511102 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:43.511212 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:43.511415 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.512204 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.513596 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.514356 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.514429 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:43.514523 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:43.514930 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 19:11:43.514977 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":81,"threads_started":1}
I20260620 19:11:43.515300 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:43.510538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:33763 (local address 127.0.0.1:42374)
0620 19:11:43.510844 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:43.510876 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:43.510943 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:43.511285 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:43.511286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:43.511288 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:43.511334 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:43.511336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.512299 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.512300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:43.513508 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:43.513510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.514562 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.514563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:43.514631 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:43.514636 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:11:43.514687 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:11:43.514728 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":86,"threads_started":1}
I20260620 19:11:43.515410 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:43.510513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:43491 (local address 127.0.0.1:57314)
0620 19:11:43.510812 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:43.510869 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:43.510944 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:43.511154 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:43.511157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:43.511160 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:43.511234 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:43.511236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.512631 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.512632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:43.513680 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:43.513683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.514514 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.514515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:43.514566 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:43.514569 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:11:43.514621 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 19:11:43.514654 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":59,"threads_started":1}
I20260620 19:11:43.515774 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 7371c68711cd4c56b7c7384313307130 (DEFAULT_TABLE table=test-table [id=a7105b93afa74cde94115c778c3fab9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:43.516028 18716 tablet_service.cc:1467] Processing CreateTablet for tablet 7371c68711cd4c56b7c7384313307130 (DEFAULT_TABLE table=test-table [id=a7105b93afa74cde94115c778c3fab9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:43.516157 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7371c68711cd4c56b7c7384313307130. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:43.516358 18716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7371c68711cd4c56b7c7384313307130. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:43.519701 18797 tablet_bootstrap.cc:492] T 7371c68711cd4c56b7c7384313307130 P f3df6b12575447ba9df78816a9c2604a: Bootstrap starting.
I20260620 19:11:43.520068 18585 tablet_service.cc:1467] Processing CreateTablet for tablet 7371c68711cd4c56b7c7384313307130 (DEFAULT_TABLE table=test-table [id=a7105b93afa74cde94115c778c3fab9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:43.520324 18797 tablet_bootstrap.cc:654] T 7371c68711cd4c56b7c7384313307130 P f3df6b12575447ba9df78816a9c2604a: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:43.520411 18585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7371c68711cd4c56b7c7384313307130. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:43.520642 18797 log.cc:824] T 7371c68711cd4c56b7c7384313307130 P f3df6b12575447ba9df78816a9c2604a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:43.520956 18796 tablet_bootstrap.cc:492] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f: Bootstrap starting.
I20260620 19:11:43.521404 18797 tablet_bootstrap.cc:492] T 7371c68711cd4c56b7c7384313307130 P f3df6b12575447ba9df78816a9c2604a: No bootstrap required, opened a new log
I20260620 19:11:43.521478 18797 ts_tablet_manager.cc:1397] T 7371c68711cd4c56b7c7384313307130 P f3df6b12575447ba9df78816a9c2604a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:11:43.521549 18796 tablet_bootstrap.cc:654] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:43.522061 18796 log.cc:824] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:43.522737 18796 tablet_bootstrap.cc:492] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f: No bootstrap required, opened a new log
I20260620 19:11:43.522812 18796 ts_tablet_manager.cc:1397] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:11:43.523953 18800 tablet_bootstrap.cc:492] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c: Bootstrap starting.
I20260620 19:11:43.524194 18797 raft_consensus.cc:348] T 7371c68711cd4c56b7c7384313307130 P f3df6b12575447ba9df78816a9c2604a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed254c0cd8434b82a07d405b2aec7e7f" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43491 } } peers { permanent_uuid: "f3df6b12575447ba9df78816a9c2604a" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40535 } } peers { permanent_uuid: "b085d0a44ada4537a5eb886e7c26ac0c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33763 } }
I20260620 19:11:43.524339 18797 raft_consensus.cc:374] T 7371c68711cd4c56b7c7384313307130 P f3df6b12575447ba9df78816a9c2604a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:43.524392 18797 raft_consensus.cc:729] T 7371c68711cd4c56b7c7384313307130 P f3df6b12575447ba9df78816a9c2604a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3df6b12575447ba9df78816a9c2604a, State: Initialized, Role: FOLLOWER
I20260620 19:11:43.524544 18797 consensus_queue.cc:260] T 7371c68711cd4c56b7c7384313307130 P f3df6b12575447ba9df78816a9c2604a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed254c0cd8434b82a07d405b2aec7e7f" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43491 } } peers { permanent_uuid: "f3df6b12575447ba9df78816a9c2604a" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40535 } } peers { permanent_uuid: "b085d0a44ada4537a5eb886e7c26ac0c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33763 } }
I20260620 19:11:43.524780 18800 tablet_bootstrap.cc:654] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:43.524796 18797 ts_tablet_manager.cc:1428] T 7371c68711cd4c56b7c7384313307130 P f3df6b12575447ba9df78816a9c2604a: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 19:11:43.524930 18520 heartbeater.cc:503] Master 127.0.203.62:45707 was elected leader, sending a full tablet report...
I20260620 19:11:43.525123 18800 log.cc:824] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:43.525270 18796 raft_consensus.cc:348] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed254c0cd8434b82a07d405b2aec7e7f" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43491 } } peers { permanent_uuid: "f3df6b12575447ba9df78816a9c2604a" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40535 } } peers { permanent_uuid: "b085d0a44ada4537a5eb886e7c26ac0c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33763 } }
I20260620 19:11:43.525465 18796 raft_consensus.cc:374] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:43.525501 18796 raft_consensus.cc:729] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed254c0cd8434b82a07d405b2aec7e7f, State: Initialized, Role: FOLLOWER
I20260620 19:11:43.525609 18796 consensus_queue.cc:260] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed254c0cd8434b82a07d405b2aec7e7f" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43491 } } peers { permanent_uuid: "f3df6b12575447ba9df78816a9c2604a" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40535 } } peers { permanent_uuid: "b085d0a44ada4537a5eb886e7c26ac0c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33763 } }
I20260620 19:11:43.525992 18782 heartbeater.cc:503] Master 127.0.203.62:45707 was elected leader, sending a full tablet report...
I20260620 19:11:43.526162 18800 tablet_bootstrap.cc:492] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c: No bootstrap required, opened a new log
I20260620 19:11:43.526199 18796 ts_tablet_manager.cc:1428] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:11:43.526233 18800 ts_tablet_manager.cc:1397] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:11:43.528343 18800 raft_consensus.cc:348] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed254c0cd8434b82a07d405b2aec7e7f" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43491 } } peers { permanent_uuid: "f3df6b12575447ba9df78816a9c2604a" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40535 } } peers { permanent_uuid: "b085d0a44ada4537a5eb886e7c26ac0c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33763 } }
I20260620 19:11:43.528483 18800 raft_consensus.cc:374] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:43.528505 18800 raft_consensus.cc:729] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b085d0a44ada4537a5eb886e7c26ac0c, State: Initialized, Role: FOLLOWER
I20260620 19:11:43.528625 18800 consensus_queue.cc:260] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed254c0cd8434b82a07d405b2aec7e7f" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43491 } } peers { permanent_uuid: "f3df6b12575447ba9df78816a9c2604a" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40535 } } peers { permanent_uuid: "b085d0a44ada4537a5eb886e7c26ac0c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33763 } }
I20260620 19:11:43.528879 18800 ts_tablet_manager.cc:1428] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:11:43.528940 18651 heartbeater.cc:503] Master 127.0.203.62:45707 was elected leader, sending a full tablet report...
W20260620 19:11:43.616411 18652 tablet.cc:2369] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:11:43.650339 18803 raft_consensus.cc:482] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:11:43.650481 18803 raft_consensus.cc:504] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed254c0cd8434b82a07d405b2aec7e7f" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43491 } } peers { permanent_uuid: "f3df6b12575447ba9df78816a9c2604a" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40535 } } peers { permanent_uuid: "b085d0a44ada4537a5eb886e7c26ac0c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33763 } }
I20260620 19:11:43.650889 18803 leader_election.cc:288] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3df6b12575447ba9df78816a9c2604a (127.0.203.1:40535), b085d0a44ada4537a5eb886e7c26ac0c (127.0.203.2:33763)
I20260620 19:11:43.653941 18793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:43.651253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:60347 (local address 127.0.203.2:33763)
0620 19:11:43.651320 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:43.651322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:43.651331 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:11:43.651363 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:43.651363 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:43.651374 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:43.651446 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:43.651611 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.652145 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.653200 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.653577 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.653663 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:43.653673 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:43.653833 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 19:11:43.653874 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:11:43.653941 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:43.651106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:33763 (local address 127.0.203.3:60347)
0620 19:11:43.651175 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:43.651189 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:43.651273 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:43.651451 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:43.651452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:43.651453 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:43.651531 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:43.651533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.652237 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.652238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:43.653103 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:43.653105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.653665 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.653666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:43.653740 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:43.653743 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:11:43.653821 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 19:11:43.653865 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":30}
I20260620 19:11:43.654063 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:43.651114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:40535 (local address 127.0.203.3:49709)
0620 19:11:43.651328 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:43.651337 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:43.651436 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:43.651637 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:43.651637 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:43.651639 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:43.651708 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:43.651709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.652361 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.652362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:43.653232 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:43.653234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.653775 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.653776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:43.653860 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:43.653862 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:11:43.653921 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 19:11:43.653958 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":4,"thread_start_us":96,"threads_started":1}
I20260620 19:11:43.654050 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:43.651224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:49709 (local address 127.0.203.1:40535)
0620 19:11:43.651276 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:43.651278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:43.651431 (+ 153us) server_negotiation.cc:407] Connection header received
0620 19:11:43.651556 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:43.651557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:43.651568 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:43.651633 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:43.651788 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.652281 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.653333 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.653691 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.653774 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:43.653783 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:43.653938 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 19:11:43.653986 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 19:11:43.654431 18605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7371c68711cd4c56b7c7384313307130" candidate_uuid: "ed254c0cd8434b82a07d405b2aec7e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b085d0a44ada4537a5eb886e7c26ac0c" is_pre_election: true
I20260620 19:11:43.654431 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7371c68711cd4c56b7c7384313307130" candidate_uuid: "ed254c0cd8434b82a07d405b2aec7e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3df6b12575447ba9df78816a9c2604a" is_pre_election: true
I20260620 19:11:43.654592 18605 raft_consensus.cc:2454] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed254c0cd8434b82a07d405b2aec7e7f in term 0.
I20260620 19:11:43.654595 18474 raft_consensus.cc:2454] T 7371c68711cd4c56b7c7384313307130 P f3df6b12575447ba9df78816a9c2604a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed254c0cd8434b82a07d405b2aec7e7f in term 0.
I20260620 19:11:43.654830 18670 leader_election.cc:302] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b085d0a44ada4537a5eb886e7c26ac0c, ed254c0cd8434b82a07d405b2aec7e7f; no voters:
I20260620 19:11:43.654989 18803 raft_consensus.cc:2781] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:11:43.655068 18803 raft_consensus.cc:482] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:11:43.655098 18803 raft_consensus.cc:3037] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:43.656036 18803 raft_consensus.cc:504] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed254c0cd8434b82a07d405b2aec7e7f" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43491 } } peers { permanent_uuid: "f3df6b12575447ba9df78816a9c2604a" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40535 } } peers { permanent_uuid: "b085d0a44ada4537a5eb886e7c26ac0c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33763 } }
I20260620 19:11:43.656194 18803 leader_election.cc:288] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [CANDIDATE]: Term 1 election: Requested vote from peers f3df6b12575447ba9df78816a9c2604a (127.0.203.1:40535), b085d0a44ada4537a5eb886e7c26ac0c (127.0.203.2:33763)
I20260620 19:11:43.656390 18605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7371c68711cd4c56b7c7384313307130" candidate_uuid: "ed254c0cd8434b82a07d405b2aec7e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b085d0a44ada4537a5eb886e7c26ac0c"
I20260620 19:11:43.656503 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7371c68711cd4c56b7c7384313307130" candidate_uuid: "ed254c0cd8434b82a07d405b2aec7e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3df6b12575447ba9df78816a9c2604a"
I20260620 19:11:43.656497 18605 raft_consensus.cc:3037] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:43.656606 18474 raft_consensus.cc:3037] T 7371c68711cd4c56b7c7384313307130 P f3df6b12575447ba9df78816a9c2604a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:43.657325 18605 raft_consensus.cc:2454] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed254c0cd8434b82a07d405b2aec7e7f in term 1.
I20260620 19:11:43.657460 18474 raft_consensus.cc:2454] T 7371c68711cd4c56b7c7384313307130 P f3df6b12575447ba9df78816a9c2604a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed254c0cd8434b82a07d405b2aec7e7f in term 1.
I20260620 19:11:43.657567 18670 leader_election.cc:302] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b085d0a44ada4537a5eb886e7c26ac0c, ed254c0cd8434b82a07d405b2aec7e7f; no voters:
I20260620 19:11:43.657692 18803 raft_consensus.cc:2781] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:43.657871 18803 raft_consensus.cc:686] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [term 1 LEADER]: Becoming Leader. State: Replica: ed254c0cd8434b82a07d405b2aec7e7f, State: Running, Role: LEADER
I20260620 19:11:43.657979 18803 consensus_queue.cc:237] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed254c0cd8434b82a07d405b2aec7e7f" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43491 } } peers { permanent_uuid: "f3df6b12575447ba9df78816a9c2604a" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40535 } } peers { permanent_uuid: "b085d0a44ada4537a5eb886e7c26ac0c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33763 } }
I20260620 19:11:43.658835 18331 catalog_manager.cc:5495] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f reported cstate change: term changed from 0 to 1, leader changed from <none> to ed254c0cd8434b82a07d405b2aec7e7f (127.0.203.3). New cstate: current_term: 1 leader_uuid: "ed254c0cd8434b82a07d405b2aec7e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed254c0cd8434b82a07d405b2aec7e7f" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3df6b12575447ba9df78816a9c2604a" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b085d0a44ada4537a5eb886e7c26ac0c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33763 } health_report { overall_health: UNKNOWN } } }
I20260620 19:11:43.696285 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:43.693860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57328 (local address 127.0.203.3:43491)
0620 19:11:43.693940 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:43.693942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:43.693950 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:11:43.693991 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:43.693992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:43.694010 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:43.694083 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:11:43.694206 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.694746 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.695564 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:43.695762 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:43.695827 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:43.695964 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:11:43.696067 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:11:43.696129 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:43.696161 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 19:11:43.696181 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:11:43.698268 18474 raft_consensus.cc:1261] T 7371c68711cd4c56b7c7384313307130 P f3df6b12575447ba9df78816a9c2604a [term 1 FOLLOWER]: Refusing update from remote peer ed254c0cd8434b82a07d405b2aec7e7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:11:43.698273 18605 raft_consensus.cc:1261] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c [term 1 FOLLOWER]: Refusing update from remote peer ed254c0cd8434b82a07d405b2aec7e7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:11:43.698688 18806 consensus_queue.cc:1035] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3df6b12575447ba9df78816a9c2604a" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:11:43.698841 18803 consensus_queue.cc:1035] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b085d0a44ada4537a5eb886e7c26ac0c" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:11:43.702011 18811 mvcc.cc:205] Tried to move back new op lower bound from 7299001154345476096 to 7299001154183979008. Current Snapshot: MvccSnapshot[applied={T|T < 7299001154345476096}]
I20260620 19:11:43.703939 18331 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:11:43.704039 18331 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:11:43.704831 18331 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a273ce5f79541b4b113eb5b3b79a39d: Sending DeleteTablet for 3 replicas of tablet 7371c68711cd4c56b7c7384313307130
I20260620 19:11:43.705191 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 7371c68711cd4c56b7c7384313307130 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:43 UTC) from {username='slave'} at 127.0.0.1:55406
I20260620 19:11:43.705193 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet 7371c68711cd4c56b7c7384313307130 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:43 UTC) from {username='slave'} at 127.0.0.1:42374
I20260620 19:11:43.705581 18820 tablet_replica.cc:330] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c: stopping tablet replica
I20260620 19:11:43.705567 18716 tablet_service.cc:1514] Processing DeleteTablet for tablet 7371c68711cd4c56b7c7384313307130 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:43 UTC) from {username='slave'} at 127.0.0.1:57314
I20260620 19:11:43.705691 18820 raft_consensus.cc:2229] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:11:43.705793 18820 raft_consensus.cc:2258] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:11:43.705823 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 18392
I20260620 19:11:43.707770 18820 ts_tablet_manager.cc:1905] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:11:43.709791 18820 ts_tablet_manager.cc:1918] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:11:43.709877 18820 log.cc:1196] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/7371c68711cd4c56b7c7384313307130
I20260620 19:11:43.710124 18820 ts_tablet_manager.cc:1939] T 7371c68711cd4c56b7c7384313307130 P b085d0a44ada4537a5eb886e7c26ac0c: Deleting consensus metadata
I20260620 19:11:43.710588 18317 catalog_manager.cc:4841] TS b085d0a44ada4537a5eb886e7c26ac0c (127.0.203.2:33763): tablet 7371c68711cd4c56b7c7384313307130 (table test-table [id=a7105b93afa74cde94115c778c3fab9c]) successfully deleted
I20260620 19:11:43.711303 18821 tablet_replica.cc:330] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f: stopping tablet replica
I20260620 19:11:43.711421 18821 raft_consensus.cc:2229] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:11:43.711550 18821 raft_consensus.cc:2258] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:11:43.712023 18821 ts_tablet_manager.cc:1905] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:11:43.712414 18318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.1:40535 (error 108)
I20260620 19:11:43.712641 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 18523
I20260620 19:11:43.712983 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:43.712861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:40535 (local address 127.0.0.1:55414)
0620 19:11:43.712910 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:43.712944 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.1:40535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 19:11:43.713181 18318 catalog_manager.cc:4584] TS f3df6b12575447ba9df78816a9c2604a (127.0.203.1:40535): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7371c68711cd4c56b7c7384313307130: Network error: Client connection negotiation failed: client connection to 127.0.203.1:40535: connect: Connection refused (error 111)
I20260620 19:11:43.713589 18821 ts_tablet_manager.cc:1918] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:11:43.713660 18821 log.cc:1196] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/7371c68711cd4c56b7c7384313307130
I20260620 19:11:43.713912 18821 ts_tablet_manager.cc:1939] T 7371c68711cd4c56b7c7384313307130 P ed254c0cd8434b82a07d405b2aec7e7f: Deleting consensus metadata
I20260620 19:11:43.714300 18318 catalog_manager.cc:4841] TS ed254c0cd8434b82a07d405b2aec7e7f (127.0.203.3:43491): tablet 7371c68711cd4c56b7c7384313307130 (table test-table [id=a7105b93afa74cde94115c778c3fab9c]) successfully deleted
I20260620 19:11:43.718703 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 18654
I20260620 19:11:43.723419 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 18301
2026-06-20T19:11:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (814 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:11:43 dist-test-slave-wd30 krb5kdc[18825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:11:43 dist-test-slave-wd30 krb5kdc[18825](info): set up 2 sockets
Jun 20 19:11:43 dist-test-slave-wd30 krb5kdc[18825](info): commencing operation
krb5kdc: starting...
W20260620 19:11:44.766968 812 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:11:44 dist-test-slave-wd30 krb5kdc[18825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:11:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:11:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:11:44.885838 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:42787
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:42787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:44.966781 18838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:44.967002 18838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:44.967025 18838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:44.968211 18838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:11:44.968255 18838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:44.968271 18838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:44.968283 18838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:11:44.968339 18838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:11:44.969815 18838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:42787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:42787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:44.970077 18838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:44.970300 18838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:44.975251 18843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:44.975260 18844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:44.975449 18838 server_base.cc:1034] running on GCE node
W20260620 19:11:44.975327 18846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:44.975792 18838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:44.976102 18838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:44.977274 18838 hybrid_clock.cc:648] HybridClock initialized: now 1781982704977265 us; error 46 us; skew 500 ppm
Jun 20 19:11:44 dist-test-slave-wd30 krb5kdc[18825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:44.980019 18838 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:11:44.980536 18838 webserver.cc:460] Webserver started at http://127.0.203.62:33787/ using document root <none> and password file <none>
I20260620 19:11:44.980767 18838 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:44.980836 18838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:44.980989 18838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:44.982059 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "71d43b4194684f97af8dd290cb5f1e94"
format_stamp: "Formatted at 2026-06-20 19:11:44 on dist-test-slave-wd30"
server_key: "183dca59a8e0d7900ffb2a91ddc2bcf2"
server_key_iv: "29075e8f88811da3848a609a7f954ef9"
server_key_version: "encryptionkey@0"
I20260620 19:11:44.982434 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "71d43b4194684f97af8dd290cb5f1e94"
format_stamp: "Formatted at 2026-06-20 19:11:44 on dist-test-slave-wd30"
server_key: "183dca59a8e0d7900ffb2a91ddc2bcf2"
server_key_iv: "29075e8f88811da3848a609a7f954ef9"
server_key_version: "encryptionkey@0"
I20260620 19:11:44.983850 18838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:11:44.984584 18853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:44.984822 18838 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:11:44.984934 18838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "71d43b4194684f97af8dd290cb5f1e94"
format_stamp: "Formatted at 2026-06-20 19:11:44 on dist-test-slave-wd30"
server_key: "183dca59a8e0d7900ffb2a91ddc2bcf2"
server_key_iv: "29075e8f88811da3848a609a7f954ef9"
server_key_version: "encryptionkey@0"
I20260620 19:11:44.985020 18838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:45.006104 18838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:45.008301 18838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:45.008476 18838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:45.012909 18838 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:42787
I20260620 19:11:45.012948 18905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:42787 every 8 connection(s)
I20260620 19:11:45.013329 18838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:11:45.014245 18906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:45.016542 18906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94: Bootstrap starting.
I20260620 19:11:45.017128 18906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:45.017416 18906 log.cc:824] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:45.018154 18906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94: No bootstrap required, opened a new log
I20260620 19:11:45.019824 18906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d43b4194684f97af8dd290cb5f1e94" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42787 } }
I20260620 19:11:45.019965 18906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:45.020017 18906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d43b4194684f97af8dd290cb5f1e94, State: Initialized, Role: FOLLOWER
I20260620 19:11:45.020169 18906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d43b4194684f97af8dd290cb5f1e94" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42787 } }
I20260620 19:11:45.020229 18906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:11:45.020251 18906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:11:45.020328 18906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:45.020872 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 18838
I20260620 19:11:45.020906 18906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d43b4194684f97af8dd290cb5f1e94" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42787 } }
I20260620 19:11:45.020969 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:11:45.021041 18906 leader_election.cc:302] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71d43b4194684f97af8dd290cb5f1e94; no voters:
I20260620 19:11:45.021222 812 external_mini_cluster.cc:1428] Setting key 3217e07382cafdba25d100bbf7e896d8
I20260620 19:11:45.021250 18906 leader_election.cc:288] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:11:45.021278 18909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:45.021440 18909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94 [term 1 LEADER]: Becoming Leader. State: Replica: 71d43b4194684f97af8dd290cb5f1e94, State: Running, Role: LEADER
I20260620 19:11:45.021512 18906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:11:45.021577 18909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d43b4194684f97af8dd290cb5f1e94" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42787 } }
I20260620 19:11:45.021929 18909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71d43b4194684f97af8dd290cb5f1e94. Latest consensus state: current_term: 1 leader_uuid: "71d43b4194684f97af8dd290cb5f1e94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d43b4194684f97af8dd290cb5f1e94" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42787 } } }
I20260620 19:11:45.022037 18909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94 [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:45.022086 18911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71d43b4194684f97af8dd290cb5f1e94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d43b4194684f97af8dd290cb5f1e94" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42787 } } }
I20260620 19:11:45.022209 18911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94 [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:45.022315 18918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:11:45.022869 18918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:11:45.024844 18918 catalog_manager.cc:1347] Generated new cluster ID: 4088cdc577654374986145875c7dbf91
I20260620 19:11:45.024909 18918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:11:45 dist-test-slave-wd30 krb5kdc[18825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:11:45.035219 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.022062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35420 (local address 127.0.203.62:42787)
0620 19:11:45.023951 (+ 1889us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:45.023960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:45.023990 (+ 30us) server_negotiation.cc:407] Connection header received
0620 19:11:45.024097 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:45.024103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:45.024145 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:45.024305 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:45.025140 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.026559 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.027711 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.028335 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.028425 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.031395 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:45.031401 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:45.031402 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:45.031417 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:45.033470 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:45.033868 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:45.033872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:45.033872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:45.034006 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:45.034219 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:45.034220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:45.034220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:45.034547 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:45.034710 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:45.034790 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 19:11:45.034935 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1838,"thread_start_us":116,"threads_started":1}
I20260620 19:11:45.041574 18918 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:11:45.042279 18918 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:11:45.047791 18918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94: Generated new TSK 0
I20260620 19:11:45.047988 18918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
I20260620 19:11:45.076020 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:42787
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:45.160701 18932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:45.160933 18932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:45.160959 18932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:45.162133 18932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:45.162179 18932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:45.162206 18932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:11:45.163807 18932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:42787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:45.164055 18932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:45.164311 18932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:45.169376 18940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:45.169346 18937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:45.169344 18938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:45.169543 18932 server_base.cc:1034] running on GCE node
I20260620 19:11:45.169808 18932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:45.170023 18932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:45.171190 18932 hybrid_clock.cc:648] HybridClock initialized: now 1781982705171180 us; error 33 us; skew 500 ppm
Jun 20 19:11:45 dist-test-slave-wd30 krb5kdc[18825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:45.173866 18932 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:11:45.174325 18932 webserver.cc:460] Webserver started at http://127.0.203.1:37547/ using document root <none> and password file <none>
I20260620 19:11:45.174574 18932 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:45.174644 18932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:45.174790 18932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:45.175932 18932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "54bda3d36c7945d09dca6564bdd82c62"
format_stamp: "Formatted at 2026-06-20 19:11:45 on dist-test-slave-wd30"
server_key: "897862469357dfdc4df920d6570f788b"
server_key_iv: "9acfe0e7b01fe50197474722121dc653"
server_key_version: "encryptionkey@0"
I20260620 19:11:45.176302 18932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "54bda3d36c7945d09dca6564bdd82c62"
format_stamp: "Formatted at 2026-06-20 19:11:45 on dist-test-slave-wd30"
server_key: "897862469357dfdc4df920d6570f788b"
server_key_iv: "9acfe0e7b01fe50197474722121dc653"
server_key_version: "encryptionkey@0"
I20260620 19:11:45.177628 18932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:11:45.178390 18947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:45.178572 18932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:11:45.178697 18932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "54bda3d36c7945d09dca6564bdd82c62"
format_stamp: "Formatted at 2026-06-20 19:11:45 on dist-test-slave-wd30"
server_key: "897862469357dfdc4df920d6570f788b"
server_key_iv: "9acfe0e7b01fe50197474722121dc653"
server_key_version: "encryptionkey@0"
I20260620 19:11:45.178781 18932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:45.191648 18932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:45.193962 18932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:45.194126 18932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:45.194367 18932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:45.194682 18932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:45.194739 18932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:45.194780 18932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:45.194808 18932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:45.200908 18932 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:43893
I20260620 19:11:45.200984 19060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:43893 every 8 connection(s)
I20260620 19:11:45.201309 18932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:11:45 dist-test-slave-wd30 krb5kdc[18825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:11:45.210837 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.202010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:43243 (local address 127.0.203.62:42787)
0620 19:11:45.202072 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:45.202074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:45.202264 (+ 190us) server_negotiation.cc:407] Connection header received
0620 19:11:45.203324 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:45.203325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:45.203339 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:45.203404 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:45.204041 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.204557 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.205541 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.205740 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.205800 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.208027 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:45.208029 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:45.208029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:45.208036 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:45.209302 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:45.209699 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:45.209699 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:45.209700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:45.209752 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:45.210005 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:45.210006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:45.210007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:45.210152 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:45.210230 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:45.210630 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 19:11:45.210734 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:11:45.211027 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.201948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42787 (local address 127.0.203.1:43243)
0620 19:11:45.202128 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:45.202169 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:45.203206 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:45.203480 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:45.203484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:45.203738 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:45.203922 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:45.203926 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.204644 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.204645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:45.205440 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:45.205442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.205818 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.205819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:45.205914 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.206435 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:11:45.206444 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:45.207927 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:45.209378 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:45.209384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:45.209385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:45.209632 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:45.209822 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:45.209823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:45.209823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:45.209937 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:45.210238 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:45.210243 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:45.210431 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 19:11:45.210575 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 19:11:45.210782 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":112,"threads_started":1}
I20260620 19:11:45.211565 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 18932
I20260620 19:11:45.211611 19061 heartbeater.cc:347] Connected to a master server at 127.0.203.62:42787
I20260620 19:11:45.211689 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:11:45.211736 19061 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:45.211925 812 external_mini_cluster.cc:1428] Setting key a352486cb97df5f667d30afc7d2552a1
I20260620 19:11:45.211932 19061 heartbeater.cc:511] Master 127.0.203.62:42787 requested a full tablet report, sending...
I20260620 19:11:45.212509 18870 ts_manager.cc:194] Registered new tserver with Master: 54bda3d36c7945d09dca6564bdd82c62 (127.0.203.1:43893)
I20260620 19:11:45.213445 18870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:43243
WARNING: no policy specified for kudu/127.0.203.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
I20260620 19:11:45.237680 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:42787
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:45.319013 19066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:45.319339 19066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:45.319370 19066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:45.320502 19066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:45.320539 19066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:45.320564 19066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:11:45.322227 19066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:42787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:45.322460 19066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:45.322692 19066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:45.328210 19074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:45.328274 19072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:45.328442 19066 server_base.cc:1034] running on GCE node
W20260620 19:11:45.328461 19071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:45.328779 19066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:45.329051 19066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:45.330242 19066 hybrid_clock.cc:648] HybridClock initialized: now 1781982705330228 us; error 31 us; skew 500 ppm
Jun 20 19:11:45 dist-test-slave-wd30 krb5kdc[18825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:45.332875 19066 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:11:45.333364 19066 webserver.cc:460] Webserver started at http://127.0.203.2:38305/ using document root <none> and password file <none>
I20260620 19:11:45.333606 19066 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:45.333657 19066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:45.333810 19066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:45.334870 19066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "52d705635b654eab8d8c9a6c374b21c3"
format_stamp: "Formatted at 2026-06-20 19:11:45 on dist-test-slave-wd30"
server_key: "287264b0810864dd8282e7601400eaa7"
server_key_iv: "175fb687b6c674d99c2d1e8303d30559"
server_key_version: "encryptionkey@0"
I20260620 19:11:45.335268 19066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "52d705635b654eab8d8c9a6c374b21c3"
format_stamp: "Formatted at 2026-06-20 19:11:45 on dist-test-slave-wd30"
server_key: "287264b0810864dd8282e7601400eaa7"
server_key_iv: "175fb687b6c674d99c2d1e8303d30559"
server_key_version: "encryptionkey@0"
I20260620 19:11:45.336588 19066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:11:45.337355 19081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:45.337539 19066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:11:45.337628 19066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "52d705635b654eab8d8c9a6c374b21c3"
format_stamp: "Formatted at 2026-06-20 19:11:45 on dist-test-slave-wd30"
server_key: "287264b0810864dd8282e7601400eaa7"
server_key_iv: "175fb687b6c674d99c2d1e8303d30559"
server_key_version: "encryptionkey@0"
I20260620 19:11:45.337684 19066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:45.358951 19066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:45.361279 19066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:45.361395 19066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:45.361617 19066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:45.361972 19066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:45.362015 19066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:45.362040 19066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:45.362093 19066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:45.367754 19066 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:38535
I20260620 19:11:45.367818 19194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:38535 every 8 connection(s)
I20260620 19:11:45.368151 19066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:11:45.372900 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 19066
I20260620 19:11:45.373066 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:11:45.373332 812 external_mini_cluster.cc:1428] Setting key 02584e9aab224ef7a8a8cd4a3e2ac08d
Jun 20 19:11:45 dist-test-slave-wd30 krb5kdc[18825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:11:45.378304 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.368905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:42995 (local address 127.0.203.62:42787)
0620 19:11:45.368954 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:45.368956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:45.369204 (+ 248us) server_negotiation.cc:407] Connection header received
0620 19:11:45.370182 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:45.370184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:45.370196 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:45.370237 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:45.370879 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.371458 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.372486 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.372678 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.372823 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.375357 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:45.375358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:45.375359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:45.375366 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:45.376817 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:45.377313 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:45.377313 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:45.377314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:45.377366 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:45.377660 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:45.377661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:45.377661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:45.377766 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:45.377821 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:45.378172 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 19:11:45.378239 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:11:45.378633 19197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.368857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42787 (local address 127.0.203.2:42995)
0620 19:11:45.369062 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:45.369099 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:45.370056 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:45.370322 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:45.370326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:45.370584 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:45.370777 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:45.370782 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.371566 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.371567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:45.372383 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:45.372385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.372832 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.372833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:45.372967 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.373642 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:11:45.373653 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:45.375238 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:45.376903 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:45.376910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:45.376912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:45.377233 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:45.377432 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:45.377433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:45.377433 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:45.377589 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:45.377838 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:45.377846 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:45.378027 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 19:11:45.378151 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 19:11:45.378338 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":102,"threads_started":1}
I20260620 19:11:45.379216 19195 heartbeater.cc:347] Connected to a master server at 127.0.203.62:42787
I20260620 19:11:45.379334 19195 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:45.379571 19195 heartbeater.cc:511] Master 127.0.203.62:42787 requested a full tablet report, sending...
I20260620 19:11:45.380023 18870 ts_manager.cc:194] Registered new tserver with Master: 52d705635b654eab8d8c9a6c374b21c3 (127.0.203.2:38535)
I20260620 19:11:45.380416 18870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:42995
WARNING: no policy specified for kudu/127.0.203.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
I20260620 19:11:45.401826 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:42787
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:45.485657 19200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:45.485880 19200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:45.485906 19200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:45.487107 19200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:45.487171 19200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:45.487198 19200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:11:45.488770 19200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:42787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:45.489017 19200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:45.489262 19200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:45.494345 19208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:45.494369 19206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:45.494362 19205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:45.494642 19200 server_base.cc:1034] running on GCE node
I20260620 19:11:45.494874 19200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:45.495221 19200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:45.496407 19200 hybrid_clock.cc:648] HybridClock initialized: now 1781982705496397 us; error 40 us; skew 500 ppm
Jun 20 19:11:45 dist-test-slave-wd30 krb5kdc[18825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:45.499368 19200 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:11:45.499892 19200 webserver.cc:460] Webserver started at http://127.0.203.3:45115/ using document root <none> and password file <none>
I20260620 19:11:45.500133 19200 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:45.500211 19200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:45.500370 19200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:45.501487 19200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "edc72620ad644ca9a193b867e61b43fc"
format_stamp: "Formatted at 2026-06-20 19:11:45 on dist-test-slave-wd30"
server_key: "0f349e35de1e41a57b7022289ac8c005"
server_key_iv: "6c838da5f501c260fd18cde31a129a9d"
server_key_version: "encryptionkey@0"
I20260620 19:11:45.501885 19200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "edc72620ad644ca9a193b867e61b43fc"
format_stamp: "Formatted at 2026-06-20 19:11:45 on dist-test-slave-wd30"
server_key: "0f349e35de1e41a57b7022289ac8c005"
server_key_iv: "6c838da5f501c260fd18cde31a129a9d"
server_key_version: "encryptionkey@0"
I20260620 19:11:45.503285 19200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:11:45.504098 19215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:45.504278 19200 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:11:45.504379 19200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "edc72620ad644ca9a193b867e61b43fc"
format_stamp: "Formatted at 2026-06-20 19:11:45 on dist-test-slave-wd30"
server_key: "0f349e35de1e41a57b7022289ac8c005"
server_key_iv: "6c838da5f501c260fd18cde31a129a9d"
server_key_version: "encryptionkey@0"
I20260620 19:11:45.504464 19200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:45.519747 19200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:45.522297 19200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:45.522502 19200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:45.522778 19200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:45.523072 19200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:45.523108 19200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:45.523182 19200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:45.523211 19200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:45.529260 19200 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:44099
I20260620 19:11:45.529304 19328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:44099 every 8 connection(s)
I20260620 19:11:45.529633 19200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:11:45 dist-test-slave-wd30 krb5kdc[18825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:11:45.537365 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 19200
I20260620 19:11:45.537488 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:11:45.537744 812 external_mini_cluster.cc:1428] Setting key 251eb41ff4346b8f515a0802b0e2ea2f
I20260620 19:11:45.539929 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.530374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:51071 (local address 127.0.203.62:42787)
0620 19:11:45.530439 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:45.530440 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:45.530648 (+ 208us) server_negotiation.cc:407] Connection header received
0620 19:11:45.531680 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:45.531682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:45.531693 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:45.531762 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:45.532405 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.532902 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.533917 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.534132 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.534193 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.536498 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:45.536500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:45.536500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:45.536508 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:45.538491 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:45.538977 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:45.538978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:45.538978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:45.539052 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:45.539326 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:45.539327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:45.539327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:45.539438 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:45.539491 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:45.539802 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 19:11:45.539866 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:11:45.540156 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.530341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:42787 (local address 127.0.203.3:51071)
0620 19:11:45.530516 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:45.530561 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:45.531573 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:45.531823 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:45.531827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:45.532099 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:45.532297 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:45.532301 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.533013 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.533013 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:45.533837 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:45.533839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.534214 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.534215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:45.534322 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.534905 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:11:45.534915 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:45.536400 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:45.538570 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:45.538577 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:45.538579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:45.538902 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:45.539108 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:45.539109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:45.539109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:45.539267 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:45.539538 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:45.539545 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:45.539674 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 19:11:45.539766 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 19:11:45.539900 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260620 19:11:45.540640 19329 heartbeater.cc:347] Connected to a master server at 127.0.203.62:42787
I20260620 19:11:45.540756 19329 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:45.540963 19329 heartbeater.cc:511] Master 127.0.203.62:42787 requested a full tablet report, sending...
I20260620 19:11:45.541414 18870 ts_manager.cc:194] Registered new tserver with Master: edc72620ad644ca9a193b867e61b43fc (127.0.203.3:44099)
I20260620 19:11:45.541818 18870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:51071
I20260620 19:11:45.549753 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:11:45.557399 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.551138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35428 (local address 127.0.203.62:42787)
0620 19:11:45.551194 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:45.551196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:45.551286 (+ 90us) server_negotiation.cc:407] Connection header received
0620 19:11:45.551378 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:45.551379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:45.551389 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:45.551463 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:45.552064 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.552557 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.553290 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.553490 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.553575 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.554915 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:45.554917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:45.554917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:45.554925 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:45.556132 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:45.556537 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:45.556538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:45.556539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:45.556600 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:45.556819 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:45.556821 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:45.556821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:45.556965 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:45.557097 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:45.557229 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 19:11:45.557295 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:11:45.559185 18870 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35428:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:11:45.560829 18870 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:11:45.567608 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.563153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:44099 (local address 127.0.0.1:58708)
0620 19:11:45.563314 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:45.563354 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:45.563421 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:45.563999 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:45.563999 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:45.564002 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:45.564050 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:45.564052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.565386 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.565387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:45.566341 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:45.566343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.567276 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.567277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:45.567361 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.567378 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 19:11:45.567450 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 19:11:45.567505 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260620 19:11:45.567616 19342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.563247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58708 (local address 127.0.203.3:44099)
0620 19:11:45.563788 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:45.563792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:45.563801 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:11:45.563863 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:45.563865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:45.563883 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:45.564134 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:45.564221 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.565268 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.566430 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.567187 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.567266 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.567332 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:45.567474 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 19:11:45.567513 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":73,"threads_started":1}
I20260620 19:11:45.567791 19345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.563793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43424 (local address 127.0.203.1:43893)
0620 19:11:45.563983 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:45.563988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:45.564064 (+ 76us) server_negotiation.cc:407] Connection header received
0620 19:11:45.564108 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:45.564110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:45.564126 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:45.564198 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:45.564392 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.565063 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.566733 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.567311 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.567376 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.567431 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:45.567652 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 19:11:45.567693 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":79,"threads_started":1}
I20260620 19:11:45.567817 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.563513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32870 (local address 127.0.203.2:38535)
0620 19:11:45.563679 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:45.563684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:45.563692 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:11:45.563746 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:45.563748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:45.563765 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:45.563855 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:45.564139 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.565249 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.566583 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.567278 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.567336 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.567382 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:45.567665 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 19:11:45.567751 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260620 19:11:45.567919 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.563361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:43893 (local address 127.0.0.1:43424)
0620 19:11:45.563478 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:45.563491 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:45.563542 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:45.564284 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:45.564284 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:45.564286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:45.564320 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:45.564321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.565161 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.565162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:45.566656 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:45.566658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.567465 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.567466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:45.567518 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.567522 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:11:45.567569 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 19:11:45.567605 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":23,"thread_start_us":60,"threads_started":1}
I20260620 19:11:45.568005 19341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.563149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:38535 (local address 127.0.0.1:32870)
0620 19:11:45.563314 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:45.563354 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:45.563421 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:45.563968 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:45.563971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:45.563974 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:45.564050 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:45.564053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.565353 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.565354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:45.566492 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:45.566494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.567442 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.567444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:45.567515 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.567519 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:11:45.567573 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 19:11:45.567622 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":9,"thread_start_us":63,"threads_started":1}
I20260620 19:11:45.568849 18995 tablet_service.cc:1467] Processing CreateTablet for tablet 7154804e30a347e5b76150cb109f3664 (DEFAULT_TABLE table=test-table [id=b01791cfacb348c3889d61d43c888cea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:45.569216 18995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7154804e30a347e5b76150cb109f3664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:45.571626 19263 tablet_service.cc:1467] Processing CreateTablet for tablet 7154804e30a347e5b76150cb109f3664 (DEFAULT_TABLE table=test-table [id=b01791cfacb348c3889d61d43c888cea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:45.571957 19346 tablet_bootstrap.cc:492] T 7154804e30a347e5b76150cb109f3664 P 54bda3d36c7945d09dca6564bdd82c62: Bootstrap starting.
I20260620 19:11:45.571951 19263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7154804e30a347e5b76150cb109f3664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:45.572577 19346 tablet_bootstrap.cc:654] T 7154804e30a347e5b76150cb109f3664 P 54bda3d36c7945d09dca6564bdd82c62: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:45.573240 19346 log.cc:824] T 7154804e30a347e5b76150cb109f3664 P 54bda3d36c7945d09dca6564bdd82c62: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:45.573972 19346 tablet_bootstrap.cc:492] T 7154804e30a347e5b76150cb109f3664 P 54bda3d36c7945d09dca6564bdd82c62: No bootstrap required, opened a new log
I20260620 19:11:45.574039 19346 ts_tablet_manager.cc:1397] T 7154804e30a347e5b76150cb109f3664 P 54bda3d36c7945d09dca6564bdd82c62: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:11:45.575169 19348 tablet_bootstrap.cc:492] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc: Bootstrap starting.
I20260620 19:11:45.575590 19129 tablet_service.cc:1467] Processing CreateTablet for tablet 7154804e30a347e5b76150cb109f3664 (DEFAULT_TABLE table=test-table [id=b01791cfacb348c3889d61d43c888cea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:45.575855 19348 tablet_bootstrap.cc:654] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:45.575922 19129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7154804e30a347e5b76150cb109f3664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:45.576093 19346 raft_consensus.cc:348] T 7154804e30a347e5b76150cb109f3664 P 54bda3d36c7945d09dca6564bdd82c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc72620ad644ca9a193b867e61b43fc" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44099 } } peers { permanent_uuid: "52d705635b654eab8d8c9a6c374b21c3" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 38535 } } peers { permanent_uuid: "54bda3d36c7945d09dca6564bdd82c62" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43893 } }
I20260620 19:11:45.576318 19346 raft_consensus.cc:374] T 7154804e30a347e5b76150cb109f3664 P 54bda3d36c7945d09dca6564bdd82c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:45.576355 19346 raft_consensus.cc:729] T 7154804e30a347e5b76150cb109f3664 P 54bda3d36c7945d09dca6564bdd82c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54bda3d36c7945d09dca6564bdd82c62, State: Initialized, Role: FOLLOWER
I20260620 19:11:45.576435 19348 log.cc:824] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:45.576510 19346 consensus_queue.cc:260] T 7154804e30a347e5b76150cb109f3664 P 54bda3d36c7945d09dca6564bdd82c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc72620ad644ca9a193b867e61b43fc" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44099 } } peers { permanent_uuid: "52d705635b654eab8d8c9a6c374b21c3" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 38535 } } peers { permanent_uuid: "54bda3d36c7945d09dca6564bdd82c62" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43893 } }
I20260620 19:11:45.576784 19346 ts_tablet_manager.cc:1428] T 7154804e30a347e5b76150cb109f3664 P 54bda3d36c7945d09dca6564bdd82c62: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 19:11:45.576923 19061 heartbeater.cc:503] Master 127.0.203.62:42787 was elected leader, sending a full tablet report...
I20260620 19:11:45.577450 19348 tablet_bootstrap.cc:492] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc: No bootstrap required, opened a new log
I20260620 19:11:45.577551 19348 ts_tablet_manager.cc:1397] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:11:45.579051 19351 tablet_bootstrap.cc:492] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3: Bootstrap starting.
I20260620 19:11:45.579237 19348 raft_consensus.cc:348] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc72620ad644ca9a193b867e61b43fc" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44099 } } peers { permanent_uuid: "52d705635b654eab8d8c9a6c374b21c3" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 38535 } } peers { permanent_uuid: "54bda3d36c7945d09dca6564bdd82c62" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43893 } }
I20260620 19:11:45.579427 19348 raft_consensus.cc:374] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:45.579473 19348 raft_consensus.cc:729] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc72620ad644ca9a193b867e61b43fc, State: Initialized, Role: FOLLOWER
I20260620 19:11:45.579608 19348 consensus_queue.cc:260] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc72620ad644ca9a193b867e61b43fc" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44099 } } peers { permanent_uuid: "52d705635b654eab8d8c9a6c374b21c3" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 38535 } } peers { permanent_uuid: "54bda3d36c7945d09dca6564bdd82c62" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43893 } }
I20260620 19:11:45.579779 19351 tablet_bootstrap.cc:654] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:45.579919 19329 heartbeater.cc:503] Master 127.0.203.62:42787 was elected leader, sending a full tablet report...
I20260620 19:11:45.579922 19348 ts_tablet_manager.cc:1428] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 19:11:45.580343 19351 log.cc:824] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:45.581209 19351 tablet_bootstrap.cc:492] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3: No bootstrap required, opened a new log
I20260620 19:11:45.581312 19351 ts_tablet_manager.cc:1397] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:11:45.582780 19351 raft_consensus.cc:348] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc72620ad644ca9a193b867e61b43fc" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44099 } } peers { permanent_uuid: "52d705635b654eab8d8c9a6c374b21c3" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 38535 } } peers { permanent_uuid: "54bda3d36c7945d09dca6564bdd82c62" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43893 } }
I20260620 19:11:45.582928 19351 raft_consensus.cc:374] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:45.582960 19351 raft_consensus.cc:729] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52d705635b654eab8d8c9a6c374b21c3, State: Initialized, Role: FOLLOWER
I20260620 19:11:45.583105 19351 consensus_queue.cc:260] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc72620ad644ca9a193b867e61b43fc" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44099 } } peers { permanent_uuid: "52d705635b654eab8d8c9a6c374b21c3" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 38535 } } peers { permanent_uuid: "54bda3d36c7945d09dca6564bdd82c62" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43893 } }
I20260620 19:11:45.583442 19351 ts_tablet_manager.cc:1428] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:11:45.583467 19195 heartbeater.cc:503] Master 127.0.203.62:42787 was elected leader, sending a full tablet report...
I20260620 19:11:45.597913 19352 raft_consensus.cc:482] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:11:45.598035 19352 raft_consensus.cc:504] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc72620ad644ca9a193b867e61b43fc" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44099 } } peers { permanent_uuid: "52d705635b654eab8d8c9a6c374b21c3" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 38535 } } peers { permanent_uuid: "54bda3d36c7945d09dca6564bdd82c62" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43893 } }
I20260620 19:11:45.598385 19352 leader_election.cc:288] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52d705635b654eab8d8c9a6c374b21c3 (127.0.203.2:38535), 54bda3d36c7945d09dca6564bdd82c62 (127.0.203.1:43893)
I20260620 19:11:45.601415 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.598600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:44597 (local address 127.0.203.2:38535)
0620 19:11:45.598674 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:45.598676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:45.598688 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:11:45.598750 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:45.598751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:45.598763 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:45.598822 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:45.599014 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.599653 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.600638 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.601046 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.601100 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.601109 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:45.601293 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 19:11:45.601339 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:11:45.601444 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.598531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:38535 (local address 127.0.203.3:44597)
0620 19:11:45.598607 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:45.598625 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:45.598683 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:45.598846 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:45.598847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:45.598849 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:45.598939 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:45.598941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.599743 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.599744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:45.600544 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:45.600547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.601135 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.601136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:45.601222 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.601225 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:11:45.601268 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 19:11:45.601303 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 19:11:45.601490 19345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.598649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:56177 (local address 127.0.203.1:43893)
0620 19:11:45.598695 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:45.598697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:45.598870 (+ 173us) server_negotiation.cc:407] Connection header received
0620 19:11:45.598956 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:45.598957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:45.598971 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:45.599023 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:45.599224 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.599758 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.600704 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.601065 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.601157 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.601166 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:45.601367 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 19:11:45.601415 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 19:11:45.601568 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.598574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:43893 (local address 127.0.203.3:56177)
0620 19:11:45.598780 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:45.598788 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:45.598895 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:45.599026 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:45.599027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:45.599029 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:45.599084 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:45.599085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.599843 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.599844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:45.600613 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:45.600616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.601184 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.601190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:45.601278 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.601284 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:11:45.601341 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 19:11:45.601377 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":144,"threads_started":1}
I20260620 19:11:45.601891 19015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7154804e30a347e5b76150cb109f3664" candidate_uuid: "edc72620ad644ca9a193b867e61b43fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54bda3d36c7945d09dca6564bdd82c62" is_pre_election: true
I20260620 19:11:45.601887 19149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7154804e30a347e5b76150cb109f3664" candidate_uuid: "edc72620ad644ca9a193b867e61b43fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52d705635b654eab8d8c9a6c374b21c3" is_pre_election: true
I20260620 19:11:45.602051 19015 raft_consensus.cc:2454] T 7154804e30a347e5b76150cb109f3664 P 54bda3d36c7945d09dca6564bdd82c62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc72620ad644ca9a193b867e61b43fc in term 0.
I20260620 19:11:45.602051 19149 raft_consensus.cc:2454] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc72620ad644ca9a193b867e61b43fc in term 0.
I20260620 19:11:45.602281 19217 leader_election.cc:302] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54bda3d36c7945d09dca6564bdd82c62, edc72620ad644ca9a193b867e61b43fc; no voters:
I20260620 19:11:45.602416 19352 raft_consensus.cc:2781] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:11:45.602471 19352 raft_consensus.cc:482] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:11:45.602501 19352 raft_consensus.cc:3037] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:45.603200 19352 raft_consensus.cc:504] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc72620ad644ca9a193b867e61b43fc" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44099 } } peers { permanent_uuid: "52d705635b654eab8d8c9a6c374b21c3" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 38535 } } peers { permanent_uuid: "54bda3d36c7945d09dca6564bdd82c62" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43893 } }
I20260620 19:11:45.603340 19352 leader_election.cc:288] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [CANDIDATE]: Term 1 election: Requested vote from peers 52d705635b654eab8d8c9a6c374b21c3 (127.0.203.2:38535), 54bda3d36c7945d09dca6564bdd82c62 (127.0.203.1:43893)
I20260620 19:11:45.603515 19149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7154804e30a347e5b76150cb109f3664" candidate_uuid: "edc72620ad644ca9a193b867e61b43fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52d705635b654eab8d8c9a6c374b21c3"
I20260620 19:11:45.603528 19015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7154804e30a347e5b76150cb109f3664" candidate_uuid: "edc72620ad644ca9a193b867e61b43fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54bda3d36c7945d09dca6564bdd82c62"
I20260620 19:11:45.603592 19149 raft_consensus.cc:3037] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:45.603638 19015 raft_consensus.cc:3037] T 7154804e30a347e5b76150cb109f3664 P 54bda3d36c7945d09dca6564bdd82c62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:45.604279 19149 raft_consensus.cc:2454] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc72620ad644ca9a193b867e61b43fc in term 1.
I20260620 19:11:45.604472 19015 raft_consensus.cc:2454] T 7154804e30a347e5b76150cb109f3664 P 54bda3d36c7945d09dca6564bdd82c62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc72620ad644ca9a193b867e61b43fc in term 1.
I20260620 19:11:45.604485 19219 leader_election.cc:302] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52d705635b654eab8d8c9a6c374b21c3, edc72620ad644ca9a193b867e61b43fc; no voters:
I20260620 19:11:45.604578 19352 raft_consensus.cc:2781] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:45.604758 19352 raft_consensus.cc:686] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [term 1 LEADER]: Becoming Leader. State: Replica: edc72620ad644ca9a193b867e61b43fc, State: Running, Role: LEADER
I20260620 19:11:45.604862 19352 consensus_queue.cc:237] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc72620ad644ca9a193b867e61b43fc" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44099 } } peers { permanent_uuid: "52d705635b654eab8d8c9a6c374b21c3" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 38535 } } peers { permanent_uuid: "54bda3d36c7945d09dca6564bdd82c62" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43893 } }
I20260620 19:11:45.605717 18870 catalog_manager.cc:5495] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc reported cstate change: term changed from 0 to 1, leader changed from <none> to edc72620ad644ca9a193b867e61b43fc (127.0.203.3). New cstate: current_term: 1 leader_uuid: "edc72620ad644ca9a193b867e61b43fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc72620ad644ca9a193b867e61b43fc" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 44099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52d705635b654eab8d8c9a6c374b21c3" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 38535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54bda3d36c7945d09dca6564bdd82c62" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43893 } health_report { overall_health: UNKNOWN } } }
I20260620 19:11:45.612986 19342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.610436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.0.203.3:44099)
0620 19:11:45.610504 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:45.610507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:45.610544 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:11:45.610630 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:45.610630 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:45.610654 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:45.610728 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:11:45.610869 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.611392 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.612172 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.612372 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.612461 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.612639 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:11:45.612736 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:11:45.612845 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:45.612875 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 19:11:45.612884 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:11:45.614651 19149 raft_consensus.cc:1261] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3 [term 1 FOLLOWER]: Refusing update from remote peer edc72620ad644ca9a193b867e61b43fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:11:45.614668 19015 raft_consensus.cc:1261] T 7154804e30a347e5b76150cb109f3664 P 54bda3d36c7945d09dca6564bdd82c62 [term 1 FOLLOWER]: Refusing update from remote peer edc72620ad644ca9a193b867e61b43fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:11:45.615007 19356 consensus_queue.cc:1035] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "52d705635b654eab8d8c9a6c374b21c3" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 38535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:11:45.615157 19352 consensus_queue.cc:1035] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "54bda3d36c7945d09dca6564bdd82c62" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:11:45.617738 19359 mvcc.cc:205] Tried to move back new op lower bound from 7299001162195226624 to 7299001162158403584. Current Snapshot: MvccSnapshot[applied={T|T < 7299001162195226624}]
I20260620 19:11:45.617946 19360 mvcc.cc:205] Tried to move back new op lower bound from 7299001162195226624 to 7299001162158403584. Current Snapshot: MvccSnapshot[applied={T|T < 7299001162195226624}]
I20260620 19:11:45.620432 18870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:11:45.620535 18870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:11:45.621344 18870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71d43b4194684f97af8dd290cb5f1e94: Sending DeleteTablet for 3 replicas of tablet 7154804e30a347e5b76150cb109f3664
I20260620 19:11:45.621739 19129 tablet_service.cc:1514] Processing DeleteTablet for tablet 7154804e30a347e5b76150cb109f3664 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:45 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:32870
I20260620 19:11:45.621830 19263 tablet_service.cc:1514] Processing DeleteTablet for tablet 7154804e30a347e5b76150cb109f3664 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:45 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:58708
I20260620 19:11:45.621975 19369 tablet_replica.cc:330] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3: stopping tablet replica
I20260620 19:11:45.621989 18995 tablet_service.cc:1514] Processing DeleteTablet for tablet 7154804e30a347e5b76150cb109f3664 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:45 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:43424
I20260620 19:11:45.622084 19369 raft_consensus.cc:2229] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:11:45.622205 19369 raft_consensus.cc:2258] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:11:45.622224 19371 tablet_replica.cc:330] T 7154804e30a347e5b76150cb109f3664 P 54bda3d36c7945d09dca6564bdd82c62: stopping tablet replica
I20260620 19:11:45.622474 19371 raft_consensus.cc:2229] T 7154804e30a347e5b76150cb109f3664 P 54bda3d36c7945d09dca6564bdd82c62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:11:45.622573 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 18932
I20260620 19:11:45.622675 19371 raft_consensus.cc:2258] T 7154804e30a347e5b76150cb109f3664 P 54bda3d36c7945d09dca6564bdd82c62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:11:45.622673 19369 ts_tablet_manager.cc:1905] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:11:45.624703 19369 ts_tablet_manager.cc:1918] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:11:45.624794 19369 log.cc:1196] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-1/wal/wals/7154804e30a347e5b76150cb109f3664
I20260620 19:11:45.625133 19369 ts_tablet_manager.cc:1939] T 7154804e30a347e5b76150cb109f3664 P 52d705635b654eab8d8c9a6c374b21c3: Deleting consensus metadata
I20260620 19:11:45.625584 18857 catalog_manager.cc:4841] TS 52d705635b654eab8d8c9a6c374b21c3 (127.0.203.2:38535): tablet 7154804e30a347e5b76150cb109f3664 (table test-table [id=b01791cfacb348c3889d61d43c888cea]) successfully deleted
I20260620 19:11:45.629148 19370 tablet_replica.cc:330] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc: stopping tablet replica
W20260620 19:11:45.629150 18855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.1:43893 (error 108)
I20260620 19:11:45.629271 19370 raft_consensus.cc:2229] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:11:45.629400 19370 raft_consensus.cc:2258] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:11:45.629745 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.629623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:43893 (local address 127.0.0.1:43436)
0620 19:11:45.629673 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:45.629709 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.1:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 19:11:45.629747 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 19066
I20260620 19:11:45.630245 19370 ts_tablet_manager.cc:1905] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:11:45.630482 18855 catalog_manager.cc:4584] TS 54bda3d36c7945d09dca6564bdd82c62 (127.0.203.1:43893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7154804e30a347e5b76150cb109f3664: Network error: Client connection negotiation failed: client connection to 127.0.203.1:43893: connect: Connection refused (error 111)
I20260620 19:11:45.632368 19370 ts_tablet_manager.cc:1918] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:11:45.632449 19370 log.cc:1196] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/7154804e30a347e5b76150cb109f3664
I20260620 19:11:45.632751 19370 ts_tablet_manager.cc:1939] T 7154804e30a347e5b76150cb109f3664 P edc72620ad644ca9a193b867e61b43fc: Deleting consensus metadata
I20260620 19:11:45.633217 18855 catalog_manager.cc:4841] TS edc72620ad644ca9a193b867e61b43fc (127.0.203.3:44099): tablet 7154804e30a347e5b76150cb109f3664 (table test-table [id=b01791cfacb348c3889d61d43c888cea]) successfully deleted
I20260620 19:11:45.636900 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 19200
I20260620 19:11:45.642221 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 18838
2026-06-20T19:11:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1919 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T19:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:11:45Z Disabled control of system clock
I20260620 19:11:45.670235 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:33481
--webserver_interface=127.0.203.62
--webserver_port=0
--builtin_ntp_servers=127.0.203.20:33935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:33481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:11:45.754068 19376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:45.754412 19376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:45.754442 19376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:45.755710 19376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:11:45.755759 19376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:45.755774 19376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:45.755789 19376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:11:45.755803 19376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:11:45.757244 19376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:33935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:33481
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:33481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:45.757543 19376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:45.757879 19376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:45.763175 19384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:45.763299 19376 server_base.cc:1034] running on GCE node
W20260620 19:11:45.763141 19382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:45.763141 19381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:45.763763 19376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:45.764055 19376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:45.765232 19376 hybrid_clock.cc:648] HybridClock initialized: now 1781982705765194 us; error 56 us; skew 500 ppm
I20260620 19:11:45.766525 19376 webserver.cc:460] Webserver started at http://127.0.203.62:44723/ using document root <none> and password file <none>
I20260620 19:11:45.766785 19376 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:45.766861 19376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:45.767020 19376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:45.768172 19376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "0f7432e7e388450895950684823dcc1a"
format_stamp: "Formatted at 2026-06-20 19:11:45 on dist-test-slave-wd30"
server_key: "e2cd1a8e6c24589c9f8dbf74df0d3927"
server_key_iv: "5b3886b0d8e213b49f425aa833626b5f"
server_key_version: "encryptionkey@0"
I20260620 19:11:45.768563 19376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "0f7432e7e388450895950684823dcc1a"
format_stamp: "Formatted at 2026-06-20 19:11:45 on dist-test-slave-wd30"
server_key: "e2cd1a8e6c24589c9f8dbf74df0d3927"
server_key_iv: "5b3886b0d8e213b49f425aa833626b5f"
server_key_version: "encryptionkey@0"
I20260620 19:11:45.769867 19376 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:11:45.770627 19390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:45.770821 19376 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:11:45.770920 19376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "0f7432e7e388450895950684823dcc1a"
format_stamp: "Formatted at 2026-06-20 19:11:45 on dist-test-slave-wd30"
server_key: "e2cd1a8e6c24589c9f8dbf74df0d3927"
server_key_iv: "5b3886b0d8e213b49f425aa833626b5f"
server_key_version: "encryptionkey@0"
I20260620 19:11:45.771005 19376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:45.783372 19376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:45.783711 19376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:45.783862 19376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:45.787974 19376 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:33481
I20260620 19:11:45.788100 19442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:33481 every 8 connection(s)
I20260620 19:11:45.788401 19376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:11:45.789381 19443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:45.791786 19443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a: Bootstrap starting.
I20260620 19:11:45.792318 19443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:45.792565 19443 log.cc:824] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:45.793470 19443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a: No bootstrap required, opened a new log
I20260620 19:11:45.794994 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 19376
I20260620 19:11:45.795153 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:11:45.795365 812 external_mini_cluster.cc:1428] Setting key c8e730a4460e72b6b5a7955ef527130d
I20260620 19:11:45.795275 19443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7432e7e388450895950684823dcc1a" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 33481 } }
I20260620 19:11:45.795421 19443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:45.795485 19443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f7432e7e388450895950684823dcc1a, State: Initialized, Role: FOLLOWER
I20260620 19:11:45.795635 19443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7432e7e388450895950684823dcc1a" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 33481 } }
I20260620 19:11:45.795710 19443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:11:45.795759 19443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:11:45.795801 19443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:45.796504 19443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7432e7e388450895950684823dcc1a" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 33481 } }
I20260620 19:11:45.796612 19443 leader_election.cc:302] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f7432e7e388450895950684823dcc1a; no voters:
I20260620 19:11:45.796759 19443 leader_election.cc:288] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:11:45.796959 19448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:45.797029 19443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:11:45.797101 19448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a [term 1 LEADER]: Becoming Leader. State: Replica: 0f7432e7e388450895950684823dcc1a, State: Running, Role: LEADER
I20260620 19:11:45.797212 19448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7432e7e388450895950684823dcc1a" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 33481 } }
I20260620 19:11:45.797585 19449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f7432e7e388450895950684823dcc1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7432e7e388450895950684823dcc1a" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 33481 } } }
I20260620 19:11:45.797677 19449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:45.797830 19450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f7432e7e388450895950684823dcc1a. Latest consensus state: current_term: 1 leader_uuid: "0f7432e7e388450895950684823dcc1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7432e7e388450895950684823dcc1a" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 33481 } } }
I20260620 19:11:45.797904 19450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:45.798034 19457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:11:45.798699 19457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:11:45.801640 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.795856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48334 (local address 127.0.203.62:33481)
0620 19:11:45.796098 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:45.796107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:45.796139 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:11:45.796267 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:45.796275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:45.796355 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:45.796514 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:45.796717 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.798258 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.799488 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.800095 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.800216 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.800913 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:45.800916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:45.800917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:45.801131 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:45.801182 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:45.801197 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:45.801266 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:45.801338 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 19:11:45.801418 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":100,"threads_started":1}
I20260620 19:11:45.802525 19457 catalog_manager.cc:1347] Generated new cluster ID: 4ef63718b04047a787f3becf7a9941e3
I20260620 19:11:45.802596 19457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:11:45.816735 19457 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:11:45.817306 19457 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:11:45.826592 19457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a: Generated new TSK 0
I20260620 19:11:45.826809 19457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:11:45.832337 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:33481
--builtin_ntp_servers=127.0.203.20:33935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:11:45.914901 19467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:45.915113 19467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:45.915184 19467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:45.916368 19467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:45.916414 19467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:45.916440 19467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:11:45.917805 19467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:33935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:33481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:45.918052 19467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:45.918355 19467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:45.924269 19475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:45.924281 19472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:45.924273 19473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:45.924523 19467 server_base.cc:1034] running on GCE node
I20260620 19:11:45.924721 19467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:45.924993 19467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:45.926155 19467 hybrid_clock.cc:648] HybridClock initialized: now 1781982705926139 us; error 33 us; skew 500 ppm
I20260620 19:11:45.930686 19467 webserver.cc:460] Webserver started at http://127.0.203.1:39553/ using document root <none> and password file <none>
I20260620 19:11:45.930905 19467 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:45.930948 19467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:45.931044 19467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:45.932204 19467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "2bd7b665b149449cb6fbf3900eb0be89"
format_stamp: "Formatted at 2026-06-20 19:11:45 on dist-test-slave-wd30"
server_key: "f4c7ab7b9311b624836296bd60d20616"
server_key_iv: "a15a5c26e792c03da736c6abfd999fea"
server_key_version: "encryptionkey@0"
I20260620 19:11:45.932528 19467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "2bd7b665b149449cb6fbf3900eb0be89"
format_stamp: "Formatted at 2026-06-20 19:11:45 on dist-test-slave-wd30"
server_key: "f4c7ab7b9311b624836296bd60d20616"
server_key_iv: "a15a5c26e792c03da736c6abfd999fea"
server_key_version: "encryptionkey@0"
I20260620 19:11:45.933861 19467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:11:45.934685 19481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:45.934900 19467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:11:45.935012 19467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "2bd7b665b149449cb6fbf3900eb0be89"
format_stamp: "Formatted at 2026-06-20 19:11:45 on dist-test-slave-wd30"
server_key: "f4c7ab7b9311b624836296bd60d20616"
server_key_iv: "a15a5c26e792c03da736c6abfd999fea"
server_key_version: "encryptionkey@0"
I20260620 19:11:45.935091 19467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:45.944967 19467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:45.945266 19467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:45.945417 19467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:45.945637 19467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:45.945953 19467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:45.946007 19467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:45.946046 19467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:45.946074 19467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:45.952066 19467 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:35591
I20260620 19:11:45.952112 19594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:35591 every 8 connection(s)
I20260620 19:11:45.952438 19467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:11:45.957633 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 19467
I20260620 19:11:45.957741 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:11:45.957733 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.953177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:52919 (local address 127.0.203.62:33481)
0620 19:11:45.953241 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:45.953243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:45.953462 (+ 219us) server_negotiation.cc:407] Connection header received
0620 19:11:45.953972 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:45.953973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:45.953987 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:45.954052 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:45.954714 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.955344 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.956304 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.956503 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.956568 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.957426 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:45.957427 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:45.957428 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:45.957434 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:45.957448 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:45.957451 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:45.957496 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:45.957612 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 19:11:45.957639 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:11:45.957937 812 external_mini_cluster.cc:1428] Setting key deed8151b93b9c0ea948bc974af82c3c
I20260620 19:11:45.957953 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:45.953137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:33481 (local address 127.0.203.1:52919)
0620 19:11:45.953327 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:45.953369 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:45.953856 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:45.954135 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:45.954139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:45.954147 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:45.954634 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:45.954637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.955426 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.955427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:45.956208 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:45.956210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:45.956587 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:45.956588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:45.956694 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:45.957267 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:11:45.957269 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:45.957302 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:11:45.957324 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:45.957517 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:45.957524 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:45.957528 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:11:45.957610 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 19:11:45.957727 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20260620 19:11:45.958444 19595 heartbeater.cc:347] Connected to a master server at 127.0.203.62:33481
I20260620 19:11:45.958549 19595 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:45.958729 19595 heartbeater.cc:511] Master 127.0.203.62:33481 requested a full tablet report, sending...
I20260620 19:11:45.959233 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:33481
--builtin_ntp_servers=127.0.203.20:33935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:11:45.959311 19406 ts_manager.cc:194] Registered new tserver with Master: 2bd7b665b149449cb6fbf3900eb0be89 (127.0.203.1:35591)
I20260620 19:11:45.959908 19406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.1:52919
W20260620 19:11:46.042737 19598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:46.042960 19598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:46.042984 19598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:46.044291 19598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:46.044337 19598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:46.044364 19598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:11:46.045708 19598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:33935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:33481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:46.045913 19598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:46.046171 19598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:46.051201 19606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:46.051203 19603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:46.051401 19598 server_base.cc:1034] running on GCE node
W20260620 19:11:46.051285 19604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:46.051667 19598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:46.051928 19598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:46.053097 19598 hybrid_clock.cc:648] HybridClock initialized: now 1781982706053086 us; error 34 us; skew 500 ppm
I20260620 19:11:46.054423 19598 webserver.cc:460] Webserver started at http://127.0.203.2:41991/ using document root <none> and password file <none>
I20260620 19:11:46.054682 19598 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:46.054754 19598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:46.054903 19598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:46.056114 19598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "a347651749ec4a4ab8ccf7a5dff265cf"
format_stamp: "Formatted at 2026-06-20 19:11:46 on dist-test-slave-wd30"
server_key: "61543f65b2eafcb0e9c49ba21347f79e"
server_key_iv: "6e6c0f827f244d6b2b1010b38c79a60f"
server_key_version: "encryptionkey@0"
I20260620 19:11:46.056502 19598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "a347651749ec4a4ab8ccf7a5dff265cf"
format_stamp: "Formatted at 2026-06-20 19:11:46 on dist-test-slave-wd30"
server_key: "61543f65b2eafcb0e9c49ba21347f79e"
server_key_iv: "6e6c0f827f244d6b2b1010b38c79a60f"
server_key_version: "encryptionkey@0"
I20260620 19:11:46.057898 19598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:11:46.058727 19612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:46.058934 19598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:11:46.059043 19598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "a347651749ec4a4ab8ccf7a5dff265cf"
format_stamp: "Formatted at 2026-06-20 19:11:46 on dist-test-slave-wd30"
server_key: "61543f65b2eafcb0e9c49ba21347f79e"
server_key_iv: "6e6c0f827f244d6b2b1010b38c79a60f"
server_key_version: "encryptionkey@0"
I20260620 19:11:46.059122 19598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:46.087605 19598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:46.087929 19598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:46.088048 19598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:46.088253 19598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:46.088625 19598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:46.088683 19598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:46.088726 19598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:46.088755 19598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:46.094630 19598 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:37081
I20260620 19:11:46.094667 19725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:37081 every 8 connection(s)
I20260620 19:11:46.095038 19598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:11:46.100497 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:46.095891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:46365 (local address 127.0.203.62:33481)
0620 19:11:46.095969 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:46.095971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:46.096229 (+ 258us) server_negotiation.cc:407] Connection header received
0620 19:11:46.096738 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:46.096740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:46.096751 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:46.096816 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:46.097619 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.098152 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.099230 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.099416 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.099479 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:46.100247 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:46.100248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:46.100248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:46.100253 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:46.100266 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:46.100270 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:46.100313 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:46.100395 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 19:11:46.100419 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:11:46.100729 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:46.095873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:33481 (local address 127.0.203.2:46365)
0620 19:11:46.096095 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:46.096138 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:46.096638 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:46.096872 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:46.096875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:46.096884 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:46.097527 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:46.097533 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.098261 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.098262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:46.099157 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:46.099160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.099498 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.099499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:46.099614 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:46.100118 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:11:46.100118 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:46.100147 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:11:46.100171 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:46.100326 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:46.100330 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:46.100333 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:11:46.100385 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 19:11:46.100468 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
I20260620 19:11:46.101289 19726 heartbeater.cc:347] Connected to a master server at 127.0.203.62:33481
I20260620 19:11:46.101398 19726 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:46.101617 19726 heartbeater.cc:511] Master 127.0.203.62:33481 requested a full tablet report, sending...
I20260620 19:11:46.102035 19406 ts_manager.cc:194] Registered new tserver with Master: a347651749ec4a4ab8ccf7a5dff265cf (127.0.203.2:37081)
I20260620 19:11:46.102424 19406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.2:46365
I20260620 19:11:46.104560 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 19598
I20260620 19:11:46.104635 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:11:46.104784 812 external_mini_cluster.cc:1428] Setting key 4b7e154f98c0d69ac3eeb188396dddb4
I20260620 19:11:46.105705 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:33481
--builtin_ntp_servers=127.0.203.20:33935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:11:46.190006 19729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:46.190250 19729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:46.190274 19729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:46.191488 19729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:46.191530 19729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:46.191555 19729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:11:46.192870 19729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:33935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:33481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:46.193063 19729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:46.193300 19729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:46.198474 19737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:46.198495 19735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:46.198536 19729 server_base.cc:1034] running on GCE node
W20260620 19:11:46.198486 19734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:46.198930 19729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:46.199190 19729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:46.200385 19729 hybrid_clock.cc:648] HybridClock initialized: now 1781982706200365 us; error 36 us; skew 500 ppm
I20260620 19:11:46.201874 19729 webserver.cc:460] Webserver started at http://127.0.203.3:38931/ using document root <none> and password file <none>
I20260620 19:11:46.202220 19729 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:46.202270 19729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:46.202423 19729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:46.203737 19729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "d8870a8dc0a0466d9879e5b652841664"
format_stamp: "Formatted at 2026-06-20 19:11:46 on dist-test-slave-wd30"
server_key: "a1b3fb168a25bda12e7a3df272823181"
server_key_iv: "e12d573fb2061090f9d15b93f5a7dff4"
server_key_version: "encryptionkey@0"
I20260620 19:11:46.204133 19729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "d8870a8dc0a0466d9879e5b652841664"
format_stamp: "Formatted at 2026-06-20 19:11:46 on dist-test-slave-wd30"
server_key: "a1b3fb168a25bda12e7a3df272823181"
server_key_iv: "e12d573fb2061090f9d15b93f5a7dff4"
server_key_version: "encryptionkey@0"
I20260620 19:11:46.205542 19729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:11:46.206339 19743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:46.206535 19729 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:11:46.206638 19729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "d8870a8dc0a0466d9879e5b652841664"
format_stamp: "Formatted at 2026-06-20 19:11:46 on dist-test-slave-wd30"
server_key: "a1b3fb168a25bda12e7a3df272823181"
server_key_iv: "e12d573fb2061090f9d15b93f5a7dff4"
server_key_version: "encryptionkey@0"
I20260620 19:11:46.206734 19729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:46.215480 19729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:46.215806 19729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:46.215978 19729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:46.216248 19729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:46.216614 19729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:46.216672 19729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:46.216724 19729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:46.216754 19729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:46.222918 19729 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:39145
I20260620 19:11:46.222993 19856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:39145 every 8 connection(s)
I20260620 19:11:46.223407 19729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:11:46.228543 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:46.224202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:37681 (local address 127.0.203.62:33481)
0620 19:11:46.224293 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:46.224295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:46.224509 (+ 214us) server_negotiation.cc:407] Connection header received
0620 19:11:46.224991 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:46.224993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:46.225003 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:46.225062 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:46.225725 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.226220 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.227277 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.227474 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.227545 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:46.228302 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:46.228303 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:46.228303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:46.228308 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:46.228323 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:46.228326 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:46.228370 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:46.228448 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 19:11:46.228474 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:11:46.228785 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:46.224157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:33481 (local address 127.0.203.3:37681)
0620 19:11:46.224380 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:46.224415 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:46.224905 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:46.225133 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:46.225137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:46.225145 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:46.225647 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:46.225651 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.226301 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.226302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:46.227182 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:46.227185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.227555 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.227555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:46.227670 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:46.228174 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:11:46.228174 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:46.228201 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:11:46.228224 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:46.228380 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:46.228384 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:46.228387 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:11:46.228438 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:11:46.228530 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":103,"threads_started":1}
I20260620 19:11:46.229346 19857 heartbeater.cc:347] Connected to a master server at 127.0.203.62:33481
I20260620 19:11:46.229458 19857 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:46.229650 19857 heartbeater.cc:511] Master 127.0.203.62:33481 requested a full tablet report, sending...
I20260620 19:11:46.230023 19406 ts_manager.cc:194] Registered new tserver with Master: d8870a8dc0a0466d9879e5b652841664 (127.0.203.3:39145)
I20260620 19:11:46.230386 19406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.3:37681
I20260620 19:11:46.230705 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 19729
I20260620 19:11:46.230809 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:11:46.230998 812 external_mini_cluster.cc:1428] Setting key 8b99d13ca00f978b045017d858a81bab
I20260620 19:11:46.231961 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:11:46.236198 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:46.233387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48342 (local address 127.0.203.62:33481)
0620 19:11:46.233491 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:46.233493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:46.233567 (+ 74us) server_negotiation.cc:407] Connection header received
0620 19:11:46.233647 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:46.233648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:46.233659 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:46.233699 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:46.233828 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.234288 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.235004 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.235243 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.235292 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:46.235920 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:46.235921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:46.235922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:46.235926 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:46.235939 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:46.235943 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:46.236019 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:46.236060 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 19:11:46.236069 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 19:11:46.237848 19406 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:11:46.239789 19406 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:11:46.246217 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:46.242056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:35591 (local address 127.0.0.1:56164)
0620 19:11:46.242383 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:46.242392 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:46.242440 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:46.242741 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:46.242744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:46.242747 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:46.242826 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:46.242828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.244173 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.244174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:46.245228 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:46.245230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.245922 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.245923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:46.245998 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:46.246011 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 19:11:46.246086 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 19:11:46.246138 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":25,"thread_start_us":93,"threads_started":1}
I20260620 19:11:46.246229 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:46.242142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56164 (local address 127.0.203.1:35591)
0620 19:11:46.242386 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:46.242391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:46.242486 (+ 95us) server_negotiation.cc:407] Connection header received
0620 19:11:46.242533 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:46.242535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:46.242552 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:46.242648 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:46.242908 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.244083 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.245309 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.245809 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.245920 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:46.246019 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:46.246099 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 19:11:46.246140 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":98,"threads_started":1}
I20260620 19:11:46.246335 19865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:46.242051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:37081 (local address 127.0.0.1:44982)
0620 19:11:46.242262 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:46.242303 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:46.242370 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:46.242744 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:46.242746 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:46.242748 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:46.242829 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:46.242832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.243836 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.243837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:46.245078 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:46.245080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.246088 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.246089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:46.246150 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:46.246154 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:11:46.246199 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 19:11:46.246243 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":7,"thread_start_us":77,"threads_started":1}
I20260620 19:11:46.246460 19866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:46.242271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:39145 (local address 127.0.0.1:49312)
0620 19:11:46.242730 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:46.242741 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:46.242790 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:46.243055 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:46.243055 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:46.243057 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:46.243089 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:46.243091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.244313 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.244314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:46.245282 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:46.245285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.246231 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.246232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:46.246279 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:46.246282 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:11:46.246327 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 19:11:46.246365 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260620 19:11:46.246676 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:46.242167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.0.203.2:37081)
0620 19:11:46.242479 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:46.242483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:46.242493 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:11:46.242537 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:46.242551 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:46.242568 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:46.242657 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:46.242912 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.243738 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.245166 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.245878 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.245963 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:46.246051 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:46.246481 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 19:11:46.246592 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":87,"threads_started":1}
I20260620 19:11:46.246675 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:46.242478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49312 (local address 127.0.203.3:39145)
0620 19:11:46.242672 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:46.242676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:46.242861 (+ 185us) server_negotiation.cc:407] Connection header received
0620 19:11:46.242908 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:46.242910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:46.242925 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:46.242993 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:46.243190 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.244231 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.245373 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.246063 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.246124 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:46.246200 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:46.246562 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 19:11:46.246598 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":96,"threads_started":1}
I20260620 19:11:46.247390 19660 tablet_service.cc:1467] Processing CreateTablet for tablet df469004f07b437094332d5e7aa13c4d (DEFAULT_TABLE table=test-table [id=ee009c4420de4f35989322ea37e5f08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:46.247572 19791 tablet_service.cc:1467] Processing CreateTablet for tablet df469004f07b437094332d5e7aa13c4d (DEFAULT_TABLE table=test-table [id=ee009c4420de4f35989322ea37e5f08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:46.247692 19529 tablet_service.cc:1467] Processing CreateTablet for tablet df469004f07b437094332d5e7aa13c4d (DEFAULT_TABLE table=test-table [id=ee009c4420de4f35989322ea37e5f08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:46.247802 19660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df469004f07b437094332d5e7aa13c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:46.247874 19791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df469004f07b437094332d5e7aa13c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:46.248005 19529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df469004f07b437094332d5e7aa13c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:46.250710 19871 tablet_bootstrap.cc:492] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664: Bootstrap starting.
I20260620 19:11:46.251389 19871 tablet_bootstrap.cc:654] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:46.251382 19872 tablet_bootstrap.cc:492] T df469004f07b437094332d5e7aa13c4d P 2bd7b665b149449cb6fbf3900eb0be89: Bootstrap starting.
I20260620 19:11:46.251714 19871 log.cc:824] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:46.251876 19872 tablet_bootstrap.cc:654] T df469004f07b437094332d5e7aa13c4d P 2bd7b665b149449cb6fbf3900eb0be89: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:46.252204 19872 log.cc:824] T df469004f07b437094332d5e7aa13c4d P 2bd7b665b149449cb6fbf3900eb0be89: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:46.252493 19873 tablet_bootstrap.cc:492] T df469004f07b437094332d5e7aa13c4d P a347651749ec4a4ab8ccf7a5dff265cf: Bootstrap starting.
I20260620 19:11:46.252594 19871 tablet_bootstrap.cc:492] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664: No bootstrap required, opened a new log
I20260620 19:11:46.252678 19871 ts_tablet_manager.cc:1397] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:11:46.253032 19873 tablet_bootstrap.cc:654] T df469004f07b437094332d5e7aa13c4d P a347651749ec4a4ab8ccf7a5dff265cf: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:46.253031 19872 tablet_bootstrap.cc:492] T df469004f07b437094332d5e7aa13c4d P 2bd7b665b149449cb6fbf3900eb0be89: No bootstrap required, opened a new log
I20260620 19:11:46.253124 19872 ts_tablet_manager.cc:1397] T df469004f07b437094332d5e7aa13c4d P 2bd7b665b149449cb6fbf3900eb0be89: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:11:46.253378 19873 log.cc:824] T df469004f07b437094332d5e7aa13c4d P a347651749ec4a4ab8ccf7a5dff265cf: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:46.255695 19873 tablet_bootstrap.cc:492] T df469004f07b437094332d5e7aa13c4d P a347651749ec4a4ab8ccf7a5dff265cf: No bootstrap required, opened a new log
I20260620 19:11:46.255788 19873 ts_tablet_manager.cc:1397] T df469004f07b437094332d5e7aa13c4d P a347651749ec4a4ab8ccf7a5dff265cf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 19:11:46.255714 19872 raft_consensus.cc:348] T df469004f07b437094332d5e7aa13c4d P 2bd7b665b149449cb6fbf3900eb0be89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd7b665b149449cb6fbf3900eb0be89" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35591 } } peers { permanent_uuid: "a347651749ec4a4ab8ccf7a5dff265cf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37081 } } peers { permanent_uuid: "d8870a8dc0a0466d9879e5b652841664" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 39145 } }
I20260620 19:11:46.255714 19871 raft_consensus.cc:348] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd7b665b149449cb6fbf3900eb0be89" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35591 } } peers { permanent_uuid: "a347651749ec4a4ab8ccf7a5dff265cf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37081 } } peers { permanent_uuid: "d8870a8dc0a0466d9879e5b652841664" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 39145 } }
I20260620 19:11:46.255889 19871 raft_consensus.cc:374] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:46.255889 19872 raft_consensus.cc:374] T df469004f07b437094332d5e7aa13c4d P 2bd7b665b149449cb6fbf3900eb0be89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:46.255934 19871 raft_consensus.cc:729] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8870a8dc0a0466d9879e5b652841664, State: Initialized, Role: FOLLOWER
I20260620 19:11:46.255940 19872 raft_consensus.cc:729] T df469004f07b437094332d5e7aa13c4d P 2bd7b665b149449cb6fbf3900eb0be89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bd7b665b149449cb6fbf3900eb0be89, State: Initialized, Role: FOLLOWER
I20260620 19:11:46.256080 19872 consensus_queue.cc:260] T df469004f07b437094332d5e7aa13c4d P 2bd7b665b149449cb6fbf3900eb0be89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd7b665b149449cb6fbf3900eb0be89" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35591 } } peers { permanent_uuid: "a347651749ec4a4ab8ccf7a5dff265cf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37081 } } peers { permanent_uuid: "d8870a8dc0a0466d9879e5b652841664" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 39145 } }
I20260620 19:11:46.256080 19871 consensus_queue.cc:260] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd7b665b149449cb6fbf3900eb0be89" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35591 } } peers { permanent_uuid: "a347651749ec4a4ab8ccf7a5dff265cf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37081 } } peers { permanent_uuid: "d8870a8dc0a0466d9879e5b652841664" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 39145 } }
I20260620 19:11:46.256350 19872 ts_tablet_manager.cc:1428] T df469004f07b437094332d5e7aa13c4d P 2bd7b665b149449cb6fbf3900eb0be89: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260620 19:11:46.256350 19871 ts_tablet_manager.cc:1428] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260620 19:11:46.256474 19595 heartbeater.cc:503] Master 127.0.203.62:33481 was elected leader, sending a full tablet report...
I20260620 19:11:46.256492 19857 heartbeater.cc:503] Master 127.0.203.62:33481 was elected leader, sending a full tablet report...
I20260620 19:11:46.258373 19873 raft_consensus.cc:348] T df469004f07b437094332d5e7aa13c4d P a347651749ec4a4ab8ccf7a5dff265cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd7b665b149449cb6fbf3900eb0be89" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35591 } } peers { permanent_uuid: "a347651749ec4a4ab8ccf7a5dff265cf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37081 } } peers { permanent_uuid: "d8870a8dc0a0466d9879e5b652841664" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 39145 } }
I20260620 19:11:46.258502 19873 raft_consensus.cc:374] T df469004f07b437094332d5e7aa13c4d P a347651749ec4a4ab8ccf7a5dff265cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:46.258525 19873 raft_consensus.cc:729] T df469004f07b437094332d5e7aa13c4d P a347651749ec4a4ab8ccf7a5dff265cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a347651749ec4a4ab8ccf7a5dff265cf, State: Initialized, Role: FOLLOWER
I20260620 19:11:46.258654 19873 consensus_queue.cc:260] T df469004f07b437094332d5e7aa13c4d P a347651749ec4a4ab8ccf7a5dff265cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd7b665b149449cb6fbf3900eb0be89" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35591 } } peers { permanent_uuid: "a347651749ec4a4ab8ccf7a5dff265cf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37081 } } peers { permanent_uuid: "d8870a8dc0a0466d9879e5b652841664" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 39145 } }
I20260620 19:11:46.258874 19873 ts_tablet_manager.cc:1428] T df469004f07b437094332d5e7aa13c4d P a347651749ec4a4ab8ccf7a5dff265cf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:11:46.259009 19726 heartbeater.cc:503] Master 127.0.203.62:33481 was elected leader, sending a full tablet report...
I20260620 19:11:46.314566 19878 raft_consensus.cc:482] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:11:46.314697 19878 raft_consensus.cc:504] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd7b665b149449cb6fbf3900eb0be89" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35591 } } peers { permanent_uuid: "a347651749ec4a4ab8ccf7a5dff265cf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37081 } } peers { permanent_uuid: "d8870a8dc0a0466d9879e5b652841664" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 39145 } }
I20260620 19:11:46.315660 19878 leader_election.cc:288] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bd7b665b149449cb6fbf3900eb0be89 (127.0.203.1:35591), a347651749ec4a4ab8ccf7a5dff265cf (127.0.203.2:37081)
I20260620 19:11:46.318176 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:46.315520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:52555 (local address 127.0.203.1:35591)
0620 19:11:46.315567 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:46.315569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:46.315581 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:11:46.315612 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:46.315613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:46.315626 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:46.315688 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:46.315867 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.316451 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.317445 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.317815 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.317867 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:46.317876 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:46.318046 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 19:11:46.318103 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:11:46.318171 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:46.315377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:35591 (local address 127.0.203.3:52555)
0620 19:11:46.315422 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:46.315435 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:46.315480 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:46.315694 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:46.315695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:46.315697 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:46.315773 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:46.315775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.316538 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.316539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:46.317353 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:46.317355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.317898 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.317899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:46.317967 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:46.317970 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:11:46.318026 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 19:11:46.318082 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":31}
I20260620 19:11:46.318220 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:46.315425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:47357 (local address 127.0.203.2:37081)
0620 19:11:46.315469 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:46.315471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:46.315737 (+ 266us) server_negotiation.cc:407] Connection header received
0620 19:11:46.315870 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:46.315871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:46.315887 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:46.315929 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:46.316073 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.316557 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.317491 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.317838 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.317927 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:46.317945 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:46.318104 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 19:11:46.318146 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:11:46.318255 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:46.315378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:37081 (local address 127.0.203.3:47357)
0620 19:11:46.315673 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:46.315683 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:46.315775 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:46.315946 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:46.315947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:46.315948 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:46.316010 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:46.316011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.316631 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.316632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:46.317365 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:46.317366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.317947 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.317948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:46.318020 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:46.318024 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:11:46.318083 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 19:11:46.318118 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":152,"thread_start_us":97,"threads_started":1}
I20260620 19:11:46.318619 19680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df469004f07b437094332d5e7aa13c4d" candidate_uuid: "d8870a8dc0a0466d9879e5b652841664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a347651749ec4a4ab8ccf7a5dff265cf" is_pre_election: true
I20260620 19:11:46.318689 19549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df469004f07b437094332d5e7aa13c4d" candidate_uuid: "d8870a8dc0a0466d9879e5b652841664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bd7b665b149449cb6fbf3900eb0be89" is_pre_election: true
I20260620 19:11:46.318766 19680 raft_consensus.cc:2454] T df469004f07b437094332d5e7aa13c4d P a347651749ec4a4ab8ccf7a5dff265cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8870a8dc0a0466d9879e5b652841664 in term 0.
I20260620 19:11:46.318859 19549 raft_consensus.cc:2454] T df469004f07b437094332d5e7aa13c4d P 2bd7b665b149449cb6fbf3900eb0be89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8870a8dc0a0466d9879e5b652841664 in term 0.
I20260620 19:11:46.319011 19747 leader_election.cc:302] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a347651749ec4a4ab8ccf7a5dff265cf, d8870a8dc0a0466d9879e5b652841664; no voters:
I20260620 19:11:46.319183 19878 raft_consensus.cc:2781] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:11:46.319239 19878 raft_consensus.cc:482] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:11:46.319262 19878 raft_consensus.cc:3037] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:46.319921 19878 raft_consensus.cc:504] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd7b665b149449cb6fbf3900eb0be89" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35591 } } peers { permanent_uuid: "a347651749ec4a4ab8ccf7a5dff265cf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37081 } } peers { permanent_uuid: "d8870a8dc0a0466d9879e5b652841664" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 39145 } }
I20260620 19:11:46.320120 19878 leader_election.cc:288] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [CANDIDATE]: Term 1 election: Requested vote from peers 2bd7b665b149449cb6fbf3900eb0be89 (127.0.203.1:35591), a347651749ec4a4ab8ccf7a5dff265cf (127.0.203.2:37081)
I20260620 19:11:46.320200 19549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df469004f07b437094332d5e7aa13c4d" candidate_uuid: "d8870a8dc0a0466d9879e5b652841664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bd7b665b149449cb6fbf3900eb0be89"
I20260620 19:11:46.320211 19680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df469004f07b437094332d5e7aa13c4d" candidate_uuid: "d8870a8dc0a0466d9879e5b652841664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a347651749ec4a4ab8ccf7a5dff265cf"
I20260620 19:11:46.320281 19680 raft_consensus.cc:3037] T df469004f07b437094332d5e7aa13c4d P a347651749ec4a4ab8ccf7a5dff265cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:46.320281 19549 raft_consensus.cc:3037] T df469004f07b437094332d5e7aa13c4d P 2bd7b665b149449cb6fbf3900eb0be89 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:46.320991 19680 raft_consensus.cc:2454] T df469004f07b437094332d5e7aa13c4d P a347651749ec4a4ab8ccf7a5dff265cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8870a8dc0a0466d9879e5b652841664 in term 1.
I20260620 19:11:46.321151 19549 raft_consensus.cc:2454] T df469004f07b437094332d5e7aa13c4d P 2bd7b665b149449cb6fbf3900eb0be89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8870a8dc0a0466d9879e5b652841664 in term 1.
I20260620 19:11:46.321161 19747 leader_election.cc:302] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a347651749ec4a4ab8ccf7a5dff265cf, d8870a8dc0a0466d9879e5b652841664; no voters:
I20260620 19:11:46.321259 19878 raft_consensus.cc:2781] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:46.321425 19878 raft_consensus.cc:686] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [term 1 LEADER]: Becoming Leader. State: Replica: d8870a8dc0a0466d9879e5b652841664, State: Running, Role: LEADER
I20260620 19:11:46.321533 19878 consensus_queue.cc:237] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd7b665b149449cb6fbf3900eb0be89" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35591 } } peers { permanent_uuid: "a347651749ec4a4ab8ccf7a5dff265cf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37081 } } peers { permanent_uuid: "d8870a8dc0a0466d9879e5b652841664" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 39145 } }
I20260620 19:11:46.322420 19407 catalog_manager.cc:5495] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8870a8dc0a0466d9879e5b652841664 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "d8870a8dc0a0466d9879e5b652841664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd7b665b149449cb6fbf3900eb0be89" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a347651749ec4a4ab8ccf7a5dff265cf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8870a8dc0a0466d9879e5b652841664" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 39145 } health_report { overall_health: HEALTHY } } }
I20260620 19:11:46.338703 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:46.336324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49328 (local address 127.0.203.3:39145)
0620 19:11:46.336371 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:46.336373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:46.336381 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:11:46.336460 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:46.336461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:46.336481 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:46.336543 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:11:46.336654 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.337112 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.337814 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:46.338005 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:46.338063 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:46.338281 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:11:46.338368 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:11:46.338513 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:46.338585 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 19:11:46.338611 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:11:46.340798 19549 raft_consensus.cc:1261] T df469004f07b437094332d5e7aa13c4d P 2bd7b665b149449cb6fbf3900eb0be89 [term 1 FOLLOWER]: Refusing update from remote peer d8870a8dc0a0466d9879e5b652841664: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:11:46.340798 19680 raft_consensus.cc:1261] T df469004f07b437094332d5e7aa13c4d P a347651749ec4a4ab8ccf7a5dff265cf [term 1 FOLLOWER]: Refusing update from remote peer d8870a8dc0a0466d9879e5b652841664: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:11:46.341195 19881 consensus_queue.cc:1035] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bd7b665b149449cb6fbf3900eb0be89" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:11:46.341354 19878 consensus_queue.cc:1035] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a347651749ec4a4ab8ccf7a5dff265cf" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 37081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:11:46.343969 19885 mvcc.cc:205] Tried to move back new op lower bound from 7299001165168771072 to 7299001165093715968. Current Snapshot: MvccSnapshot[applied={T|T < 7299001165168771072}]
I20260620 19:11:46.344591 19886 mvcc.cc:205] Tried to move back new op lower bound from 7299001165168771072 to 7299001165093715968. Current Snapshot: MvccSnapshot[applied={T|T < 7299001165168771072 or (T in {7299001165168771072})}]
I20260620 19:11:46.346639 19406 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:11:46.346740 19406 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:11:46.347597 19406 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f7432e7e388450895950684823dcc1a: Sending DeleteTablet for 3 replicas of tablet df469004f07b437094332d5e7aa13c4d
I20260620 19:11:46.347951 19791 tablet_service.cc:1514] Processing DeleteTablet for tablet df469004f07b437094332d5e7aa13c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:46 UTC) from {username='slave'} at 127.0.0.1:49312
I20260620 19:11:46.348008 19660 tablet_service.cc:1514] Processing DeleteTablet for tablet df469004f07b437094332d5e7aa13c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:46 UTC) from {username='slave'} at 127.0.0.1:44982
I20260620 19:11:46.348188 19894 tablet_replica.cc:330] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664: stopping tablet replica
I20260620 19:11:46.348218 19529 tablet_service.cc:1514] Processing DeleteTablet for tablet df469004f07b437094332d5e7aa13c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:46 UTC) from {username='slave'} at 127.0.0.1:56164
I20260620 19:11:46.348287 19894 raft_consensus.cc:2229] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:11:46.348412 19894 raft_consensus.cc:2258] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:11:46.348528 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 19467
I20260620 19:11:46.348423 19896 tablet_replica.cc:330] T df469004f07b437094332d5e7aa13c4d P 2bd7b665b149449cb6fbf3900eb0be89: stopping tablet replica
I20260620 19:11:46.348737 19896 raft_consensus.cc:2229] T df469004f07b437094332d5e7aa13c4d P 2bd7b665b149449cb6fbf3900eb0be89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:11:46.348858 19894 ts_tablet_manager.cc:1905] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:11:46.348896 19896 raft_consensus.cc:2258] T df469004f07b437094332d5e7aa13c4d P 2bd7b665b149449cb6fbf3900eb0be89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:11:46.350910 19894 ts_tablet_manager.cc:1918] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:11:46.350989 19894 log.cc:1196] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/df469004f07b437094332d5e7aa13c4d
I20260620 19:11:46.351284 19894 ts_tablet_manager.cc:1939] T df469004f07b437094332d5e7aa13c4d P d8870a8dc0a0466d9879e5b652841664: Deleting consensus metadata
W20260620 19:11:46.354719 19393 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.1:35591 (error 108)
I20260620 19:11:46.354843 19394 catalog_manager.cc:4841] TS d8870a8dc0a0466d9879e5b652841664 (127.0.203.3:39145): tablet df469004f07b437094332d5e7aa13c4d (table test-table [id=ee009c4420de4f35989322ea37e5f08c]) successfully deleted
I20260620 19:11:46.355062 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 19598
I20260620 19:11:46.355077 19895 tablet_replica.cc:330] T df469004f07b437094332d5e7aa13c4d P a347651749ec4a4ab8ccf7a5dff265cf: stopping tablet replica
I20260620 19:11:46.355321 19865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:46.355184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:35591 (local address 127.0.0.1:56180)
0620 19:11:46.355230 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:46.355278 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.1:35591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 19:11:46.355427 19895 raft_consensus.cc:2229] T df469004f07b437094332d5e7aa13c4d P a347651749ec4a4ab8ccf7a5dff265cf [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 19:11:46.357319 19393 catalog_manager.cc:4584] TS 2bd7b665b149449cb6fbf3900eb0be89 (127.0.203.1:35591): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet df469004f07b437094332d5e7aa13c4d: Network error: Client connection negotiation failed: client connection to 127.0.203.1:35591: connect: Connection refused (error 111)
I20260620 19:11:46.359265 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 19729
I20260620 19:11:46.359439 19865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:46.359303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:37081 (local address 127.0.0.1:44988)
0620 19:11:46.359375 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:46.359403 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.2:37081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 19:11:46.364852 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 19376
2026-06-20T19:11:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (721 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:11:46 dist-test-slave-wd30 krb5kdc[19900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:11:46 dist-test-slave-wd30 krb5kdc[19900](info): set up 2 sockets
Jun 20 19:11:46 dist-test-slave-wd30 krb5kdc[19900](info): commencing operation
krb5kdc: starting...
W20260620 19:11:47.406976 812 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:11:47 dist-test-slave-wd30 krb5kdc[19900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:11:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:11:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:11:47.525452 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:44901
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:44901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:47.605724 19913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:47.605928 19913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:47.605953 19913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:47.607057 19913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:11:47.607093 19913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:47.607107 19913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:47.607137 19913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:11:47.607151 19913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:11:47.608676 19913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:44901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:44901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:47.608871 19913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:47.609035 19913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:47.613940 19921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:47.613946 19918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:47.614044 19919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:47.614233 19913 server_base.cc:1034] running on GCE node
I20260620 19:11:47.614631 19913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:47.614936 19913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:47.616106 19913 hybrid_clock.cc:648] HybridClock initialized: now 1781982707616094 us; error 31 us; skew 500 ppm
Jun 20 19:11:47 dist-test-slave-wd30 krb5kdc[19900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:47.618532 19913 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:11:47.619029 19913 webserver.cc:460] Webserver started at http://127.0.203.62:40079/ using document root <none> and password file <none>
I20260620 19:11:47.619288 19913 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:47.619361 19913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:47.619508 19913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:47.620579 19913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "0f62ccf22bd94af3a352765e0075a985"
format_stamp: "Formatted at 2026-06-20 19:11:47 on dist-test-slave-wd30"
server_key: "54e38f46c950493317fd8f48df30a3ed"
server_key_iv: "b28bc1fe5238162be0bdeb6586621cec"
server_key_version: "encryptionkey@0"
I20260620 19:11:47.620980 19913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "0f62ccf22bd94af3a352765e0075a985"
format_stamp: "Formatted at 2026-06-20 19:11:47 on dist-test-slave-wd30"
server_key: "54e38f46c950493317fd8f48df30a3ed"
server_key_iv: "b28bc1fe5238162be0bdeb6586621cec"
server_key_version: "encryptionkey@0"
I20260620 19:11:47.622426 19913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:11:47.623225 19928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:47.623400 19913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:11:47.623499 19913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "0f62ccf22bd94af3a352765e0075a985"
format_stamp: "Formatted at 2026-06-20 19:11:47 on dist-test-slave-wd30"
server_key: "54e38f46c950493317fd8f48df30a3ed"
server_key_iv: "b28bc1fe5238162be0bdeb6586621cec"
server_key_version: "encryptionkey@0"
I20260620 19:11:47.623581 19913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:47.643925 19913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:47.646556 19913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:47.646749 19913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:47.651373 19913 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:44901
I20260620 19:11:47.651415 19980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:44901 every 8 connection(s)
I20260620 19:11:47.651786 19913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:11:47.652588 19981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:47.654913 19981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985: Bootstrap starting.
I20260620 19:11:47.655503 19981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:47.655799 19981 log.cc:824] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:47.656533 19981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985: No bootstrap required, opened a new log
I20260620 19:11:47.658113 19981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f62ccf22bd94af3a352765e0075a985" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44901 } }
I20260620 19:11:47.658288 19981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:47.658334 19981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f62ccf22bd94af3a352765e0075a985, State: Initialized, Role: FOLLOWER
I20260620 19:11:47.658535 19981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f62ccf22bd94af3a352765e0075a985" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44901 } }
I20260620 19:11:47.658618 19981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:11:47.658664 19981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:11:47.658723 19981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:47.659334 19981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f62ccf22bd94af3a352765e0075a985" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44901 } }
I20260620 19:11:47.659447 19981 leader_election.cc:302] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f62ccf22bd94af3a352765e0075a985; no voters:
I20260620 19:11:47.659611 19981 leader_election.cc:288] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:11:47.659657 19984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:47.659886 19981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:11:47.659890 19984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985 [term 1 LEADER]: Becoming Leader. State: Replica: 0f62ccf22bd94af3a352765e0075a985, State: Running, Role: LEADER
I20260620 19:11:47.660053 19984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f62ccf22bd94af3a352765e0075a985" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44901 } }
I20260620 19:11:47.660430 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 19913
I20260620 19:11:47.660552 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:11:47.660548 19985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f62ccf22bd94af3a352765e0075a985" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f62ccf22bd94af3a352765e0075a985" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44901 } } }
I20260620 19:11:47.660626 19985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985 [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:47.660775 812 external_mini_cluster.cc:1428] Setting key 7ec9a56ce37a63193dd7a562f51a89c7
I20260620 19:11:47.660799 19986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f62ccf22bd94af3a352765e0075a985. Latest consensus state: current_term: 1 leader_uuid: "0f62ccf22bd94af3a352765e0075a985" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f62ccf22bd94af3a352765e0075a985" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44901 } } }
I20260620 19:11:47.660867 19986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985 [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:47.661346 19993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:11:47.661890 19993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:11:47.663410 19993 catalog_manager.cc:1347] Generated new cluster ID: 9c3980b573064723909ec3e211565f54
I20260620 19:11:47.663461 19993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:11:47 dist-test-slave-wd30 krb5kdc[19900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:11:47.672745 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:47.661442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.0.203.62:44901)
0620 19:11:47.661599 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:47.661608 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:47.661646 (+ 38us) server_negotiation.cc:407] Connection header received
0620 19:11:47.661778 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:47.661786 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:47.661911 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:47.662129 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:47.662918 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:47.664320 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:47.665367 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:47.666028 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:47.666154 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:47.669017 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:47.669021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:47.669021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:47.669033 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:47.671036 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:47.671491 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:47.671494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:47.671494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:47.671605 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:47.671824 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:47.671824 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:47.671825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:47.672115 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:47.672255 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:47.672384 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 19:11:47.672517 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260620 19:11:47.674923 19993 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:11:47.675468 19993 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:11:47.679281 19993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985: Generated new TSK 0
I20260620 19:11:47.679462 19993 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
Entry for principal kudu/127.0.203.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab.
I20260620 19:11:47.710085 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44901
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:47.792136 20007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:47.792402 20007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:47.792443 20007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:47.793632 20007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:47.793702 20007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:47.793756 20007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:11:47.795315 20007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/kudu_127.0.203.1.keytab
--principal=kudu/127.0.203.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:47.795588 20007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:47.795825 20007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:11:47.801270 20007 server_base.cc:1034] running on GCE node
W20260620 19:11:47.801221 20012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:47.801221 20013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:47.801223 20015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:47.801676 20007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:47.801957 20007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:47.803112 20007 hybrid_clock.cc:648] HybridClock initialized: now 1781982707803100 us; error 27 us; skew 500 ppm
Jun 20 19:11:47 dist-test-slave-wd30 krb5kdc[19900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:47.805946 20007 init.cc:377] Logged in from keytab as kudu/127.0.203.1@KRBTEST.COM (short username kudu)
I20260620 19:11:47.806394 20007 webserver.cc:460] Webserver started at http://127.0.203.1:36165/ using document root <none> and password file <none>
I20260620 19:11:47.806651 20007 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:47.806726 20007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:47.806869 20007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:47.807967 20007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "78f2ded8f5c64d31a16d86ebfbeaa025"
format_stamp: "Formatted at 2026-06-20 19:11:47 on dist-test-slave-wd30"
server_key: "2466b78eba1146a95d6a98dc23912bd2"
server_key_iv: "2b78fff1adb3040abc9c0b131edbe343"
server_key_version: "encryptionkey@0"
I20260620 19:11:47.808326 20007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "78f2ded8f5c64d31a16d86ebfbeaa025"
format_stamp: "Formatted at 2026-06-20 19:11:47 on dist-test-slave-wd30"
server_key: "2466b78eba1146a95d6a98dc23912bd2"
server_key_iv: "2b78fff1adb3040abc9c0b131edbe343"
server_key_version: "encryptionkey@0"
I20260620 19:11:47.809679 20007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:11:47.810473 20022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:47.810648 20007 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:11:47.810750 20007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "78f2ded8f5c64d31a16d86ebfbeaa025"
format_stamp: "Formatted at 2026-06-20 19:11:47 on dist-test-slave-wd30"
server_key: "2466b78eba1146a95d6a98dc23912bd2"
server_key_iv: "2b78fff1adb3040abc9c0b131edbe343"
server_key_version: "encryptionkey@0"
I20260620 19:11:47.810833 20007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:47.824791 20007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:47.827493 20007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:47.827639 20007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:47.827843 20007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:47.828153 20007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:47.828192 20007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:47.828254 20007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:47.828284 20007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:47.833988 20007 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:42771
I20260620 19:11:47.834048 20135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:42771 every 8 connection(s)
I20260620 19:11:47.834401 20007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:11:47.835331 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 20007
I20260620 19:11:47.835443 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:11:47.835647 812 external_mini_cluster.cc:1428] Setting key 0e4c9da4903b6c837740b2f609bb01f8
Jun 20 19:11:47 dist-test-slave-wd30 krb5kdc[19900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.1@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.203.2@KRBTEST.COM; defaulting to no policy
I20260620 19:11:47.846006 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:47.835186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:48037 (local address 127.0.203.62:44901)
0620 19:11:47.835253 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:47.835255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:47.835519 (+ 264us) server_negotiation.cc:407] Connection header received
0620 19:11:47.836714 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:47.836715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:47.836728 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:47.836794 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:47.837403 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:47.838123 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:47.839762 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:47.839963 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:47.840019 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:47.842750 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:47.842752 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:47.842752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:47.842769 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:47.844604 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:47.845059 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:47.845060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:47.845060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:47.845123 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:47.845362 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:47.845362 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:47.845363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:47.845486 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:47.845540 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:47.845816 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 19:11:47.845939 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:11:47.846264 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:47.835180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:44901 (local address 127.0.203.1:48037)
0620 19:11:47.835367 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:47.835427 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:47.836580 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:47.836867 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:47.836870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:47.837119 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:47.837306 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:47.837310 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:47.838234 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:47.838244 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:47.839667 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:47.839670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:47.840065 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:47.840066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:47.840227 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:47.840794 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:11:47.840805 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:47.842573 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:47.844714 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:47.844720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:47.844722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:47.844949 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:47.845193 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:47.845193 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:47.845194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:47.845299 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:47.845544 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:47.845548 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:47.845698 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 19:11:47.845782 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:11:47.845987 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
I20260620 19:11:47.846814 20136 heartbeater.cc:347] Connected to a master server at 127.0.203.62:44901
I20260620 19:11:47.846917 20136 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:47.847078 20136 heartbeater.cc:511] Master 127.0.203.62:44901 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.2@KRBTEST.COM" created.
I20260620 19:11:47.847703 19945 ts_manager.cc:194] Registered new tserver with Master: 78f2ded8f5c64d31a16d86ebfbeaa025 (127.0.203.1:42771)
I20260620 19:11:47.848683 19945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.1@KRBTEST.COM'} at 127.0.203.1:48037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
Entry for principal kudu/127.0.203.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab.
I20260620 19:11:47.861891 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44901
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:47.941493 20141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:47.941708 20141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:47.941740 20141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:47.942934 20141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:47.942978 20141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:47.943005 20141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:11:47.944517 20141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/kudu_127.0.203.2.keytab
--principal=kudu/127.0.203.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:47.944764 20141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:47.945011 20141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:47.950016 20146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:47.950049 20147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:47.950016 20149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:47.950251 20141 server_base.cc:1034] running on GCE node
I20260620 19:11:47.950465 20141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:47.950698 20141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:47.951844 20141 hybrid_clock.cc:648] HybridClock initialized: now 1781982707951824 us; error 39 us; skew 500 ppm
Jun 20 19:11:47 dist-test-slave-wd30 krb5kdc[19900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:47.954339 20141 init.cc:377] Logged in from keytab as kudu/127.0.203.2@KRBTEST.COM (short username kudu)
I20260620 19:11:47.954777 20141 webserver.cc:460] Webserver started at http://127.0.203.2:42421/ using document root <none> and password file <none>
I20260620 19:11:47.954998 20141 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:47.955065 20141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:47.955245 20141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:47.956318 20141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "f7fda928d79d4a509e916aa2c47dfcfd"
format_stamp: "Formatted at 2026-06-20 19:11:47 on dist-test-slave-wd30"
server_key: "b0c1ea8e8165f3fb5f98c04eaebf5fbf"
server_key_iv: "0c696eacc666250c09a36def59f8164b"
server_key_version: "encryptionkey@0"
I20260620 19:11:47.956682 20141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "f7fda928d79d4a509e916aa2c47dfcfd"
format_stamp: "Formatted at 2026-06-20 19:11:47 on dist-test-slave-wd30"
server_key: "b0c1ea8e8165f3fb5f98c04eaebf5fbf"
server_key_iv: "0c696eacc666250c09a36def59f8164b"
server_key_version: "encryptionkey@0"
I20260620 19:11:47.957959 20141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:11:47.958652 20156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:47.958813 20141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:11:47.958910 20141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "f7fda928d79d4a509e916aa2c47dfcfd"
format_stamp: "Formatted at 2026-06-20 19:11:47 on dist-test-slave-wd30"
server_key: "b0c1ea8e8165f3fb5f98c04eaebf5fbf"
server_key_iv: "0c696eacc666250c09a36def59f8164b"
server_key_version: "encryptionkey@0"
I20260620 19:11:47.958990 20141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:47.969689 20141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:47.971995 20141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:47.972164 20141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:47.972394 20141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:47.972707 20141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:47.972761 20141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:47.972796 20141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:47.972823 20141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:47.978708 20141 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:44999
I20260620 19:11:47.978760 20269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:44999 every 8 connection(s)
I20260620 19:11:47.979055 20141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:11:47 dist-test-slave-wd30 krb5kdc[19900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.2@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:11:47.986563 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 20141
I20260620 19:11:47.986681 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:11:47.986905 812 external_mini_cluster.cc:1428] Setting key 9aebc0a4ab4fd9d175b2ea6484957595
I20260620 19:11:47.988292 20272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:47.979805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:44901 (local address 127.0.203.2:35755)
0620 19:11:47.979969 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:47.980004 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:47.980911 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:47.981116 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:47.981119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:47.981352 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:47.981530 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:47.981534 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:47.982193 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:47.982194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:47.982995 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:47.982997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:47.983410 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:47.983411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:47.983512 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:47.983962 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:11:47.983970 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:47.985311 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:47.986870 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:47.986876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:47.986878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:47.987193 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:47.987397 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:47.987398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:47.987398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:47.987515 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:47.987815 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:47.987820 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:47.987932 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 19:11:47.988022 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 19:11:47.988137 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260620 19:11:47.988302 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:47.979816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:35755 (local address 127.0.203.62:44901)
0620 19:11:47.979868 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:47.979871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:47.980090 (+ 219us) server_negotiation.cc:407] Connection header received
0620 19:11:47.981000 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:47.981001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:47.981011 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:47.981062 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:47.981606 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:47.982117 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:47.983071 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:47.983340 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:47.983393 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:47.985400 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:47.985401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:47.985401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:47.985407 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:47.986793 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:47.987268 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:47.987269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:47.987270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:47.987334 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:47.987592 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:47.987593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:47.987594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:47.987744 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:47.987819 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:47.988069 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 19:11:47.988166 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:11:47.988816 20270 heartbeater.cc:347] Connected to a master server at 127.0.203.62:44901
I20260620 19:11:47.988920 20270 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:47.989104 20270 heartbeater.cc:511] Master 127.0.203.62:44901 requested a full tablet report, sending...
I20260620 19:11:47.989476 19945 ts_manager.cc:194] Registered new tserver with Master: f7fda928d79d4a509e916aa2c47dfcfd (127.0.203.2:44999)
I20260620 19:11:47.989987 19945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.2@KRBTEST.COM'} at 127.0.203.2:35755
WARNING: no policy specified for kudu/127.0.203.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
Entry for principal kudu/127.0.203.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab.
I20260620 19:11:48.012010 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44901
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:48.091534 20275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:48.091745 20275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:48.091768 20275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:48.092918 20275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:48.092958 20275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:48.092984 20275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:11:48.094455 20275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/krb5kdc/kudu_127.0.203.3.keytab
--principal=kudu/127.0.203.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:48.094696 20275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:48.094937 20275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:48.099915 20280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:48.099941 20275 server_base.cc:1034] running on GCE node
W20260620 19:11:48.099956 20281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:48.099921 20283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:48.100262 20275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:48.100517 20275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:48.101691 20275 hybrid_clock.cc:648] HybridClock initialized: now 1781982708101678 us; error 26 us; skew 500 ppm
Jun 20 19:11:48 dist-test-slave-wd30 krb5kdc[19900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:48.104346 20275 init.cc:377] Logged in from keytab as kudu/127.0.203.3@KRBTEST.COM (short username kudu)
I20260620 19:11:48.104854 20275 webserver.cc:460] Webserver started at http://127.0.203.3:36543/ using document root <none> and password file <none>
I20260620 19:11:48.105077 20275 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:48.105144 20275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:48.105297 20275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:48.106345 20275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "ed8f2746b48b444db08749918f4fd780"
format_stamp: "Formatted at 2026-06-20 19:11:48 on dist-test-slave-wd30"
server_key: "440ec402f89b600f652003527a89ba18"
server_key_iv: "0a491ca812d6d681fc15b5e7276f9d74"
server_key_version: "encryptionkey@0"
I20260620 19:11:48.106734 20275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "ed8f2746b48b444db08749918f4fd780"
format_stamp: "Formatted at 2026-06-20 19:11:48 on dist-test-slave-wd30"
server_key: "440ec402f89b600f652003527a89ba18"
server_key_iv: "0a491ca812d6d681fc15b5e7276f9d74"
server_key_version: "encryptionkey@0"
I20260620 19:11:48.108114 20275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:11:48.108806 20290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:48.108973 20275 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:11:48.109071 20275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "ed8f2746b48b444db08749918f4fd780"
format_stamp: "Formatted at 2026-06-20 19:11:48 on dist-test-slave-wd30"
server_key: "440ec402f89b600f652003527a89ba18"
server_key_iv: "0a491ca812d6d681fc15b5e7276f9d74"
server_key_version: "encryptionkey@0"
I20260620 19:11:48.109156 20275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:48.118075 20275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:48.120709 20275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:48.120870 20275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:48.121126 20275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:48.121443 20275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:48.121477 20275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:48.121498 20275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:48.121511 20275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:48.127240 20275 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:35967
I20260620 19:11:48.127308 20403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:35967 every 8 connection(s)
I20260620 19:11:48.127612 20275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:11:48 dist-test-slave-wd30 krb5kdc[19900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.3@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:11:48.136350 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:48.128290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:49841 (local address 127.0.203.62:44901)
0620 19:11:48.128364 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:48.128366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:48.128570 (+ 204us) server_negotiation.cc:407] Connection header received
0620 19:11:48.129464 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:48.129465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:48.129473 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:48.129531 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:48.130082 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.130589 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.131618 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.131809 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.131863 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:48.133907 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:48.133908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:48.133908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:48.133914 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:48.135111 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:48.135587 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:48.135588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:48.135588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:48.135641 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:48.135854 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:48.135854 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:48.135855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:48.135954 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:48.136004 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:48.136223 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 19:11:48.136285 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:11:48.136566 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:48.128275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:44901 (local address 127.0.203.3:49841)
0620 19:11:48.128447 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:48.128488 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:48.129380 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:48.129584 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:48.129587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:48.129833 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:48.130015 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:48.130019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.130666 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.130666 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:48.131543 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:48.131546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.131880 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.131881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:48.131975 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:48.132498 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:11:48.132506 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:48.133823 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:48.135254 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:48.135260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:48.135262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:48.135522 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:48.135706 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:48.135707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:48.135707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:48.135799 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:48.136006 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:48.136010 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:48.136117 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 19:11:48.136194 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 19:11:48.136318 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":106,"threads_started":1}
I20260620 19:11:48.136863 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 20275
I20260620 19:11:48.136986 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:11:48.137110 20404 heartbeater.cc:347] Connected to a master server at 127.0.203.62:44901
I20260620 19:11:48.137166 812 external_mini_cluster.cc:1428] Setting key 6e24ee28d2b14a254f0a297850a39032
I20260620 19:11:48.137229 20404 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:48.137435 20404 heartbeater.cc:511] Master 127.0.203.62:44901 requested a full tablet report, sending...
I20260620 19:11:48.137820 19944 ts_manager.cc:194] Registered new tserver with Master: ed8f2746b48b444db08749918f4fd780 (127.0.203.3:35967)
I20260620 19:11:48.138221 19944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.3@KRBTEST.COM'} at 127.0.203.3:49841
I20260620 19:11:48.148907 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:11:48.156188 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:48.150333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46476 (local address 127.0.203.62:44901)
0620 19:11:48.150401 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:48.150403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:48.150530 (+ 127us) server_negotiation.cc:407] Connection header received
0620 19:11:48.150572 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:48.150573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:48.150581 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:48.150629 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:48.151265 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.151751 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.152454 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.152627 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.152675 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:48.153799 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:48.153800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:48.153800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:48.153806 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:48.155080 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:48.155438 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:48.155438 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:48.155439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:48.155487 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:48.155732 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:48.155733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:48.155733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:48.155868 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:48.155942 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:48.156059 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 19:11:48.156123 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:11:48.157910 19944 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:11:48.159554 19944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:11:48.165794 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:48.161633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:35967 (local address 127.0.0.1:40670)
0620 19:11:48.161979 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:48.161990 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:48.162045 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:48.162528 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:48.162529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:48.162531 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:48.162565 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:48.162566 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.163788 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.163789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:48.164699 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:48.164701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.165578 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.165579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:48.165641 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:48.165643 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:11:48.165702 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 19:11:48.165730 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":50,"threads_started":1}
I20260620 19:11:48.165896 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:48.161755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.0.203.1:42771)
0620 19:11:48.161974 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:48.161979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:48.161997 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:11:48.162106 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:48.162108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:48.162124 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:48.162202 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:48.162418 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.163416 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.164852 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.165338 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.165425 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:48.165484 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:48.165772 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 19:11:48.165817 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
I20260620 19:11:48.165732 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:48.161613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:42771 (local address 127.0.0.1:35654)
0620 19:11:48.161910 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:48.161950 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:48.162014 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:48.162277 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:48.162280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:48.162283 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:48.162351 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:48.162353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.163521 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.163523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:48.164783 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:48.164785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.165425 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.165426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:48.165503 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:48.165517 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 19:11:48.165582 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 19:11:48.165632 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":57,"threads_started":1}
I20260620 19:11:48.165831 20416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:48.161816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40670 (local address 127.0.203.3:35967)
0620 19:11:48.162361 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:48.162366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:48.162376 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:11:48.162414 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:48.162417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:48.162433 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:48.162628 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:48.162690 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.163694 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.164809 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.165507 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.165582 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:48.165641 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:48.165714 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 19:11:48.165751 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":84,"threads_started":1}
I20260620 19:11:48.166628 20070 tablet_service.cc:1467] Processing CreateTablet for tablet 0587130ab2124e909cde5cc4d26017a4 (DEFAULT_TABLE table=test-table [id=59a9cbdd6e4e4bcd8a412b3d514a0e21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:48.166653 20338 tablet_service.cc:1467] Processing CreateTablet for tablet 0587130ab2124e909cde5cc4d26017a4 (DEFAULT_TABLE table=test-table [id=59a9cbdd6e4e4bcd8a412b3d514a0e21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:48.166985 20070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0587130ab2124e909cde5cc4d26017a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:48.166988 20338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0587130ab2124e909cde5cc4d26017a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:48.167428 20415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:48.161808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:44999 (local address 127.0.0.1:58648)
0620 19:11:48.163191 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:48.163203 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:48.163270 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:48.163433 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:48.163433 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:48.163435 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:48.163472 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:48.163473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.164770 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.164771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:48.166038 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:48.166041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.167188 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.167189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:48.167251 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:48.167255 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:11:48.167307 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 19:11:48.167354 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1371,"thread_start_us":40,"threads_started":1}
I20260620 19:11:48.167704 20417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:48.162070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58648 (local address 127.0.203.2:44999)
0620 19:11:48.162278 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:48.162282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:48.163284 (+ 1002us) server_negotiation.cc:407] Connection header received
0620 19:11:48.163331 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:48.163333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:48.163347 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:48.163430 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:48.163531 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.164668 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.166135 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.166985 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.167055 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:48.167151 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:48.167585 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 19:11:48.167630 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":100,"threads_started":1}
I20260620 19:11:48.168351 20204 tablet_service.cc:1467] Processing CreateTablet for tablet 0587130ab2124e909cde5cc4d26017a4 (DEFAULT_TABLE table=test-table [id=59a9cbdd6e4e4bcd8a412b3d514a0e21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:48.168674 20204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0587130ab2124e909cde5cc4d26017a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:48.169974 20418 tablet_bootstrap.cc:492] T 0587130ab2124e909cde5cc4d26017a4 P 78f2ded8f5c64d31a16d86ebfbeaa025: Bootstrap starting.
I20260620 19:11:48.170578 20418 tablet_bootstrap.cc:654] T 0587130ab2124e909cde5cc4d26017a4 P 78f2ded8f5c64d31a16d86ebfbeaa025: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:48.171054 20418 log.cc:824] T 0587130ab2124e909cde5cc4d26017a4 P 78f2ded8f5c64d31a16d86ebfbeaa025: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:48.171365 20419 tablet_bootstrap.cc:492] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780: Bootstrap starting.
I20260620 19:11:48.171772 20418 tablet_bootstrap.cc:492] T 0587130ab2124e909cde5cc4d26017a4 P 78f2ded8f5c64d31a16d86ebfbeaa025: No bootstrap required, opened a new log
I20260620 19:11:48.171864 20418 ts_tablet_manager.cc:1397] T 0587130ab2124e909cde5cc4d26017a4 P 78f2ded8f5c64d31a16d86ebfbeaa025: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:11:48.172034 20419 tablet_bootstrap.cc:654] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:48.172031 20421 tablet_bootstrap.cc:492] T 0587130ab2124e909cde5cc4d26017a4 P f7fda928d79d4a509e916aa2c47dfcfd: Bootstrap starting.
I20260620 19:11:48.172518 20419 log.cc:824] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:48.172637 20421 tablet_bootstrap.cc:654] T 0587130ab2124e909cde5cc4d26017a4 P f7fda928d79d4a509e916aa2c47dfcfd: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:48.173291 20421 log.cc:824] T 0587130ab2124e909cde5cc4d26017a4 P f7fda928d79d4a509e916aa2c47dfcfd: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:48.173563 20419 tablet_bootstrap.cc:492] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780: No bootstrap required, opened a new log
I20260620 19:11:48.173650 20419 ts_tablet_manager.cc:1397] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:11:48.173920 20418 raft_consensus.cc:348] T 0587130ab2124e909cde5cc4d26017a4 P 78f2ded8f5c64d31a16d86ebfbeaa025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed8f2746b48b444db08749918f4fd780" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35967 } } peers { permanent_uuid: "78f2ded8f5c64d31a16d86ebfbeaa025" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42771 } } peers { permanent_uuid: "f7fda928d79d4a509e916aa2c47dfcfd" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44999 } }
I20260620 19:11:48.174104 20418 raft_consensus.cc:374] T 0587130ab2124e909cde5cc4d26017a4 P 78f2ded8f5c64d31a16d86ebfbeaa025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:48.174129 20421 tablet_bootstrap.cc:492] T 0587130ab2124e909cde5cc4d26017a4 P f7fda928d79d4a509e916aa2c47dfcfd: No bootstrap required, opened a new log
I20260620 19:11:48.174144 20418 raft_consensus.cc:729] T 0587130ab2124e909cde5cc4d26017a4 P 78f2ded8f5c64d31a16d86ebfbeaa025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f2ded8f5c64d31a16d86ebfbeaa025, State: Initialized, Role: FOLLOWER
I20260620 19:11:48.174191 20421 ts_tablet_manager.cc:1397] T 0587130ab2124e909cde5cc4d26017a4 P f7fda928d79d4a509e916aa2c47dfcfd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:11:48.174279 20418 consensus_queue.cc:260] T 0587130ab2124e909cde5cc4d26017a4 P 78f2ded8f5c64d31a16d86ebfbeaa025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed8f2746b48b444db08749918f4fd780" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35967 } } peers { permanent_uuid: "78f2ded8f5c64d31a16d86ebfbeaa025" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42771 } } peers { permanent_uuid: "f7fda928d79d4a509e916aa2c47dfcfd" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44999 } }
I20260620 19:11:48.174551 20418 ts_tablet_manager.cc:1428] T 0587130ab2124e909cde5cc4d26017a4 P 78f2ded8f5c64d31a16d86ebfbeaa025: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:11:48.174711 20136 heartbeater.cc:503] Master 127.0.203.62:44901 was elected leader, sending a full tablet report...
I20260620 19:11:48.175491 20419 raft_consensus.cc:348] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed8f2746b48b444db08749918f4fd780" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35967 } } peers { permanent_uuid: "78f2ded8f5c64d31a16d86ebfbeaa025" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42771 } } peers { permanent_uuid: "f7fda928d79d4a509e916aa2c47dfcfd" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44999 } }
I20260620 19:11:48.175671 20419 raft_consensus.cc:374] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:48.175714 20419 raft_consensus.cc:729] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed8f2746b48b444db08749918f4fd780, State: Initialized, Role: FOLLOWER
I20260620 19:11:48.175832 20419 consensus_queue.cc:260] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed8f2746b48b444db08749918f4fd780" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35967 } } peers { permanent_uuid: "78f2ded8f5c64d31a16d86ebfbeaa025" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42771 } } peers { permanent_uuid: "f7fda928d79d4a509e916aa2c47dfcfd" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44999 } }
I20260620 19:11:48.176110 20419 ts_tablet_manager.cc:1428] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 19:11:48.176132 20404 heartbeater.cc:503] Master 127.0.203.62:44901 was elected leader, sending a full tablet report...
I20260620 19:11:48.176285 20421 raft_consensus.cc:348] T 0587130ab2124e909cde5cc4d26017a4 P f7fda928d79d4a509e916aa2c47dfcfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed8f2746b48b444db08749918f4fd780" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35967 } } peers { permanent_uuid: "78f2ded8f5c64d31a16d86ebfbeaa025" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42771 } } peers { permanent_uuid: "f7fda928d79d4a509e916aa2c47dfcfd" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44999 } }
I20260620 19:11:48.176441 20421 raft_consensus.cc:374] T 0587130ab2124e909cde5cc4d26017a4 P f7fda928d79d4a509e916aa2c47dfcfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:48.176482 20421 raft_consensus.cc:729] T 0587130ab2124e909cde5cc4d26017a4 P f7fda928d79d4a509e916aa2c47dfcfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7fda928d79d4a509e916aa2c47dfcfd, State: Initialized, Role: FOLLOWER
I20260620 19:11:48.176586 20421 consensus_queue.cc:260] T 0587130ab2124e909cde5cc4d26017a4 P f7fda928d79d4a509e916aa2c47dfcfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed8f2746b48b444db08749918f4fd780" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35967 } } peers { permanent_uuid: "78f2ded8f5c64d31a16d86ebfbeaa025" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42771 } } peers { permanent_uuid: "f7fda928d79d4a509e916aa2c47dfcfd" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44999 } }
I20260620 19:11:48.176863 20421 ts_tablet_manager.cc:1428] T 0587130ab2124e909cde5cc4d26017a4 P f7fda928d79d4a509e916aa2c47dfcfd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:11:48.176949 20270 heartbeater.cc:503] Master 127.0.203.62:44901 was elected leader, sending a full tablet report...
W20260620 19:11:48.229663 20271 tablet.cc:2369] T 0587130ab2124e909cde5cc4d26017a4 P f7fda928d79d4a509e916aa2c47dfcfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:11:48.335204 20137 tablet.cc:2369] T 0587130ab2124e909cde5cc4d26017a4 P 78f2ded8f5c64d31a16d86ebfbeaa025: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:11:48.378158 20405 tablet.cc:2369] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:11:48.625806 20425 raft_consensus.cc:482] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:11:48.625988 20425 raft_consensus.cc:504] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed8f2746b48b444db08749918f4fd780" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35967 } } peers { permanent_uuid: "78f2ded8f5c64d31a16d86ebfbeaa025" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42771 } } peers { permanent_uuid: "f7fda928d79d4a509e916aa2c47dfcfd" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44999 } }
I20260620 19:11:48.626428 20425 leader_election.cc:288] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78f2ded8f5c64d31a16d86ebfbeaa025 (127.0.203.1:42771), f7fda928d79d4a509e916aa2c47dfcfd (127.0.203.2:44999)
I20260620 19:11:48.630529 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:48.626770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:51737 (local address 127.0.203.1:42771)
0620 19:11:48.626853 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:48.626857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:48.626872 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:11:48.626922 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:48.626924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:48.626937 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:48.627002 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:48.627258 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.628012 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.629566 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.630189 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.630255 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:48.630265 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:48.630415 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 19:11:48.630461 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 19:11:48.630555 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:48.626665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:42771 (local address 127.0.203.3:51737)
0620 19:11:48.626735 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:48.626748 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:48.626797 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:48.627048 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:48.627049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:48.627051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:48.627183 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:48.627186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.628133 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.628135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:48.629448 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:48.629452 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.630273 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.630274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:48.630331 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:48.630335 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:11:48.630405 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 19:11:48.630449 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":88}
I20260620 19:11:48.630892 20090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0587130ab2124e909cde5cc4d26017a4" candidate_uuid: "ed8f2746b48b444db08749918f4fd780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f2ded8f5c64d31a16d86ebfbeaa025" is_pre_election: true
I20260620 19:11:48.631068 20090 raft_consensus.cc:2454] T 0587130ab2124e909cde5cc4d26017a4 P 78f2ded8f5c64d31a16d86ebfbeaa025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed8f2746b48b444db08749918f4fd780 in term 0.
I20260620 19:11:48.631237 20427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:48.626698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:44999 (local address 127.0.203.3:45273)
0620 19:11:48.626837 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:48.626847 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:48.626907 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:48.627048 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:48.627049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:48.627051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:48.627183 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:48.627186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.628133 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.628135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:48.629448 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:48.629452 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.630938 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.630939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:48.631016 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:48.631020 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:11:48.631083 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 19:11:48.631151 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":19,"thread_start_us":91,"threads_started":1}
I20260620 19:11:48.631234 20417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:48.626761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:45273 (local address 127.0.203.2:44999)
0620 19:11:48.626810 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:48.626812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:48.626912 (+ 100us) server_negotiation.cc:407] Connection header received
0620 19:11:48.626960 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:48.626961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:48.626971 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:48.627039 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:48.627259 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.628020 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.630310 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.630811 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.630914 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:48.630926 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:48.631092 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 19:11:48.631169 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:11:48.631379 20294 leader_election.cc:302] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78f2ded8f5c64d31a16d86ebfbeaa025, ed8f2746b48b444db08749918f4fd780; no voters:
I20260620 19:11:48.631507 20425 raft_consensus.cc:2781] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:11:48.631562 20425 raft_consensus.cc:482] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:11:48.631583 20425 raft_consensus.cc:3037] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:48.632324 20425 raft_consensus.cc:504] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed8f2746b48b444db08749918f4fd780" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35967 } } peers { permanent_uuid: "78f2ded8f5c64d31a16d86ebfbeaa025" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42771 } } peers { permanent_uuid: "f7fda928d79d4a509e916aa2c47dfcfd" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44999 } }
I20260620 19:11:48.632478 20425 leader_election.cc:288] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [CANDIDATE]: Term 1 election: Requested vote from peers 78f2ded8f5c64d31a16d86ebfbeaa025 (127.0.203.1:42771), f7fda928d79d4a509e916aa2c47dfcfd (127.0.203.2:44999)
I20260620 19:11:48.632647 20090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0587130ab2124e909cde5cc4d26017a4" candidate_uuid: "ed8f2746b48b444db08749918f4fd780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f2ded8f5c64d31a16d86ebfbeaa025"
I20260620 19:11:48.632653 20224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0587130ab2124e909cde5cc4d26017a4" candidate_uuid: "ed8f2746b48b444db08749918f4fd780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7fda928d79d4a509e916aa2c47dfcfd" is_pre_election: true
I20260620 19:11:48.632745 20090 raft_consensus.cc:3037] T 0587130ab2124e909cde5cc4d26017a4 P 78f2ded8f5c64d31a16d86ebfbeaa025 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:48.632710 20223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0587130ab2124e909cde5cc4d26017a4" candidate_uuid: "ed8f2746b48b444db08749918f4fd780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7fda928d79d4a509e916aa2c47dfcfd"
I20260620 19:11:48.632791 20224 raft_consensus.cc:2454] T 0587130ab2124e909cde5cc4d26017a4 P f7fda928d79d4a509e916aa2c47dfcfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed8f2746b48b444db08749918f4fd780 in term 0.
I20260620 19:11:48.633526 20090 raft_consensus.cc:2454] T 0587130ab2124e909cde5cc4d26017a4 P 78f2ded8f5c64d31a16d86ebfbeaa025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed8f2746b48b444db08749918f4fd780 in term 1.
I20260620 19:11:48.633687 20294 leader_election.cc:302] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78f2ded8f5c64d31a16d86ebfbeaa025, ed8f2746b48b444db08749918f4fd780; no voters: f7fda928d79d4a509e916aa2c47dfcfd
I20260620 19:11:48.633790 20425 raft_consensus.cc:2781] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:48.633975 20425 raft_consensus.cc:686] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [term 1 LEADER]: Becoming Leader. State: Replica: ed8f2746b48b444db08749918f4fd780, State: Running, Role: LEADER
I20260620 19:11:48.634085 20425 consensus_queue.cc:237] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed8f2746b48b444db08749918f4fd780" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35967 } } peers { permanent_uuid: "78f2ded8f5c64d31a16d86ebfbeaa025" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42771 } } peers { permanent_uuid: "f7fda928d79d4a509e916aa2c47dfcfd" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44999 } }
I20260620 19:11:48.634882 19944 catalog_manager.cc:5495] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed8f2746b48b444db08749918f4fd780 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "ed8f2746b48b444db08749918f4fd780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed8f2746b48b444db08749918f4fd780" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 35967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78f2ded8f5c64d31a16d86ebfbeaa025" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7fda928d79d4a509e916aa2c47dfcfd" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44999 } health_report { overall_health: UNKNOWN } } }
I20260620 19:11:48.715234 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:48.712598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40684 (local address 127.0.203.3:35967)
0620 19:11:48.712718 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:48.712720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:48.712784 (+ 64us) server_negotiation.cc:407] Connection header received
0620 19:11:48.712843 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:48.712844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:48.712872 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:48.712943 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:11:48.713119 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.713636 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.714420 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.714665 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.714722 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:48.714860 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:11:48.714962 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:11:48.715022 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:48.715060 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 19:11:48.715076 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":52,"threads_started":1}
I20260620 19:11:48.716970 20223 raft_consensus.cc:3037] T 0587130ab2124e909cde5cc4d26017a4 P f7fda928d79d4a509e916aa2c47dfcfd [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:48.717067 20090 raft_consensus.cc:1261] T 0587130ab2124e909cde5cc4d26017a4 P 78f2ded8f5c64d31a16d86ebfbeaa025 [term 1 FOLLOWER]: Refusing update from remote peer ed8f2746b48b444db08749918f4fd780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:11:48.717408 20428 consensus_queue.cc:1035] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78f2ded8f5c64d31a16d86ebfbeaa025" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 42771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:11:48.717924 20223 raft_consensus.cc:1261] T 0587130ab2124e909cde5cc4d26017a4 P f7fda928d79d4a509e916aa2c47dfcfd [term 1 FOLLOWER]: Refusing update from remote peer ed8f2746b48b444db08749918f4fd780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:11:48.718240 20428 consensus_queue.cc:1035] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7fda928d79d4a509e916aa2c47dfcfd" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 44999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:11:48.719724 20433 mvcc.cc:205] Tried to move back new op lower bound from 7299001174902394880 to 7299001174565806080. Current Snapshot: MvccSnapshot[applied={T|T < 7299001174902394880}]
I20260620 19:11:48.721982 19944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:11:48.722096 19944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:11:48.722457 20435 mvcc.cc:205] Tried to move back new op lower bound from 7299001174902394880 to 7299001174565806080. Current Snapshot: MvccSnapshot[applied={T|T < 7299001174902394880 or (T in {7299001174902394880})}]
I20260620 19:11:48.722839 19944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f62ccf22bd94af3a352765e0075a985: Sending DeleteTablet for 3 replicas of tablet 0587130ab2124e909cde5cc4d26017a4
I20260620 19:11:48.723172 20070 tablet_service.cc:1514] Processing DeleteTablet for tablet 0587130ab2124e909cde5cc4d26017a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:48 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:35654
I20260620 19:11:48.723223 20338 tablet_service.cc:1514] Processing DeleteTablet for tablet 0587130ab2124e909cde5cc4d26017a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:48 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:40670
I20260620 19:11:48.723420 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 20007
I20260620 19:11:48.723412 20204 tablet_service.cc:1514] Processing DeleteTablet for tablet 0587130ab2124e909cde5cc4d26017a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:48 UTC) from {username='kudu', principal='kudu/127.0.203.62@KRBTEST.COM'} at 127.0.0.1:58648
I20260620 19:11:48.723454 20445 tablet_replica.cc:330] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780: stopping tablet replica
I20260620 19:11:48.723557 20445 raft_consensus.cc:2229] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:11:48.723634 20445 raft_consensus.cc:2258] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:11:48.724329 20445 ts_tablet_manager.cc:1905] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:11:48.725871 20445 ts_tablet_manager.cc:1918] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:11:48.725947 20445 log.cc:1196] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/0587130ab2124e909cde5cc4d26017a4
I20260620 19:11:48.726289 20445 ts_tablet_manager.cc:1939] T 0587130ab2124e909cde5cc4d26017a4 P ed8f2746b48b444db08749918f4fd780: Deleting consensus metadata
I20260620 19:11:48.726699 19931 catalog_manager.cc:4841] TS ed8f2746b48b444db08749918f4fd780 (127.0.203.3:35967): tablet 0587130ab2124e909cde5cc4d26017a4 (table test-table [id=59a9cbdd6e4e4bcd8a412b3d514a0e21]) successfully deleted
I20260620 19:11:48.730067 20446 tablet_replica.cc:330] T 0587130ab2124e909cde5cc4d26017a4 P f7fda928d79d4a509e916aa2c47dfcfd: stopping tablet replica
I20260620 19:11:48.730196 20446 raft_consensus.cc:2229] T 0587130ab2124e909cde5cc4d26017a4 P f7fda928d79d4a509e916aa2c47dfcfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:11:48.730326 20446 raft_consensus.cc:2258] T 0587130ab2124e909cde5cc4d26017a4 P f7fda928d79d4a509e916aa2c47dfcfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:11:48.730669 20446 ts_tablet_manager.cc:1905] T 0587130ab2124e909cde5cc4d26017a4 P f7fda928d79d4a509e916aa2c47dfcfd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:11:48.730947 19932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.1:42771 (error 108)
I20260620 19:11:48.731202 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 20141
I20260620 19:11:48.735318 20448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:48.731268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:42771 (local address 127.0.0.1:35658)
0620 19:11:48.735237 (+ 3969us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:48.735282 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.1:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3951,"thread_start_us":44,"threads_started":1}
W20260620 19:11:48.735498 19932 catalog_manager.cc:4584] TS 78f2ded8f5c64d31a16d86ebfbeaa025 (127.0.203.1:42771): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0587130ab2124e909cde5cc4d26017a4: Network error: Client connection negotiation failed: client connection to 127.0.203.1:42771: connect: Connection refused (error 111)
I20260620 19:11:48.737120 20448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:48.737000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:44999 (local address 127.0.0.1:58658)
0620 19:11:48.737062 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:48.737093 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.2:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 19:11:48.737126 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 20275
I20260620 19:11:48.742009 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 19913
2026-06-20T19:11:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2378 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T19:11:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:11:48Z Disabled control of system clock
I20260620 19:11:48.770138 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:44961
--webserver_interface=127.0.203.62
--webserver_port=0
--builtin_ntp_servers=127.0.203.20:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:44961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:11:48.850912 20452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:48.851171 20452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:48.851238 20452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:48.852406 20452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:11:48.852444 20452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:48.852458 20452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:48.852470 20452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:11:48.852483 20452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:11:48.853749 20452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:44961
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:44961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:48.854010 20452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:48.854267 20452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:48.859010 20457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:48.859031 20460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:48.859037 20458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:48.859297 20452 server_base.cc:1034] running on GCE node
I20260620 19:11:48.859560 20452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:48.859850 20452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:48.861018 20452 hybrid_clock.cc:648] HybridClock initialized: now 1781982708861002 us; error 27 us; skew 500 ppm
I20260620 19:11:48.862280 20452 webserver.cc:460] Webserver started at http://127.0.203.62:46417/ using document root <none> and password file <none>
I20260620 19:11:48.862520 20452 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:48.862599 20452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:48.862778 20452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:48.863979 20452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "6506212edf8a4419a9723c221d3937d4"
format_stamp: "Formatted at 2026-06-20 19:11:48 on dist-test-slave-wd30"
server_key: "a7f080b47f604771f2f97c634cd35dbe"
server_key_iv: "2e303b797d629df8ab00c0ed7ad8edd3"
server_key_version: "encryptionkey@0"
I20260620 19:11:48.864379 20452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "6506212edf8a4419a9723c221d3937d4"
format_stamp: "Formatted at 2026-06-20 19:11:48 on dist-test-slave-wd30"
server_key: "a7f080b47f604771f2f97c634cd35dbe"
server_key_iv: "2e303b797d629df8ab00c0ed7ad8edd3"
server_key_version: "encryptionkey@0"
I20260620 19:11:48.865659 20452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:11:48.866345 20466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:48.866513 20452 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:48.866619 20452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "6506212edf8a4419a9723c221d3937d4"
format_stamp: "Formatted at 2026-06-20 19:11:48 on dist-test-slave-wd30"
server_key: "a7f080b47f604771f2f97c634cd35dbe"
server_key_iv: "2e303b797d629df8ab00c0ed7ad8edd3"
server_key_version: "encryptionkey@0"
I20260620 19:11:48.866698 20452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:48.894644 20452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:48.894996 20452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:48.895231 20452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:48.899068 20452 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:44961
I20260620 19:11:48.899170 20518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:44961 every 8 connection(s)
I20260620 19:11:48.899508 20452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:11:48.900478 20519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:48.902793 20519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4: Bootstrap starting.
I20260620 19:11:48.903430 20519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:48.903757 20519 log.cc:824] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:48.904439 20519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4: No bootstrap required, opened a new log
I20260620 19:11:48.904788 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 20452
I20260620 19:11:48.904872 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:11:48.905084 812 external_mini_cluster.cc:1428] Setting key 8ddaaa9e554a6d5bd8d3564966f97794
I20260620 19:11:48.906677 20519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6506212edf8a4419a9723c221d3937d4" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44961 } }
I20260620 19:11:48.906855 20519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:48.906893 20519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6506212edf8a4419a9723c221d3937d4, State: Initialized, Role: FOLLOWER
I20260620 19:11:48.907047 20519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6506212edf8a4419a9723c221d3937d4" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44961 } }
I20260620 19:11:48.907174 20519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:11:48.907219 20519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:11:48.907272 20519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:48.908023 20519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6506212edf8a4419a9723c221d3937d4" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44961 } }
I20260620 19:11:48.908130 20519 leader_election.cc:302] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6506212edf8a4419a9723c221d3937d4; no voters:
I20260620 19:11:48.908301 20519 leader_election.cc:288] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:11:48.908365 20524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:48.908515 20524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4 [term 1 LEADER]: Becoming Leader. State: Replica: 6506212edf8a4419a9723c221d3937d4, State: Running, Role: LEADER
I20260620 19:11:48.908543 20519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:11:48.908625 20524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6506212edf8a4419a9723c221d3937d4" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44961 } }
I20260620 19:11:48.908953 20524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6506212edf8a4419a9723c221d3937d4. Latest consensus state: current_term: 1 leader_uuid: "6506212edf8a4419a9723c221d3937d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6506212edf8a4419a9723c221d3937d4" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44961 } } }
I20260620 19:11:48.909055 20524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4 [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:48.908962 20525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6506212edf8a4419a9723c221d3937d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6506212edf8a4419a9723c221d3937d4" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 44961 } } }
I20260620 19:11:48.909164 20525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4 [sys.catalog]: This master's current role is: LEADER
W20260620 19:11:48.910424 20540 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:11:48.910481 20540 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:11:48.910537 20531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:11:48.910688 20531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:11:48.912266 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:48.905666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.0.203.62:44961)
0620 19:11:48.905907 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:48.905916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:48.905949 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:11:48.906640 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:48.906650 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:48.906742 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:48.906895 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:48.907160 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.908973 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.910158 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:48.910646 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:48.910778 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:48.911535 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:48.911538 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:48.911539 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:48.911743 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:48.911788 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:48.911800 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:48.911884 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:48.911966 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 19:11:48.912055 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
I20260620 19:11:48.912348 20531 catalog_manager.cc:1347] Generated new cluster ID: 91b73f3abdc3413680120aa16062250e
I20260620 19:11:48.912385 20531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:11:48.920990 20531 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:11:48.921535 20531 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:11:48.926258 20531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4: Generated new TSK 0
I20260620 19:11:48.926452 20531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:11:48.932078 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.1:0
--local_ip_for_outbound_sockets=127.0.203.1
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44961
--builtin_ntp_servers=127.0.203.20:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:11:49.035822 20543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:49.036064 20543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:49.036103 20543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:49.037266 20543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:49.037329 20543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:49.037371 20543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.1
I20260620 19:11:49.038767 20543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.1
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.1
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:49.039023 20543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:49.039377 20543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:49.045131 20549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:49.045233 20551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:49.045122 20548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:49.045493 20543 server_base.cc:1034] running on GCE node
I20260620 19:11:49.045715 20543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:49.046007 20543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:49.047185 20543 hybrid_clock.cc:648] HybridClock initialized: now 1781982709047166 us; error 36 us; skew 500 ppm
I20260620 19:11:49.048532 20543 webserver.cc:460] Webserver started at http://127.0.203.1:39157/ using document root <none> and password file <none>
I20260620 19:11:49.048761 20543 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:49.048827 20543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:49.048965 20543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:49.050077 20543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-0/data/instance:
uuid: "fa2daecd7d9a4f4cb7e9fb9bc6ff43f9"
format_stamp: "Formatted at 2026-06-20 19:11:49 on dist-test-slave-wd30"
server_key: "d89173f074daf4a740f4be08aaec6ad0"
server_key_iv: "a343d03523ebba3c05d327b2e39d18b6"
server_key_version: "encryptionkey@0"
I20260620 19:11:49.050450 20543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-0/wal/instance:
uuid: "fa2daecd7d9a4f4cb7e9fb9bc6ff43f9"
format_stamp: "Formatted at 2026-06-20 19:11:49 on dist-test-slave-wd30"
server_key: "d89173f074daf4a740f4be08aaec6ad0"
server_key_iv: "a343d03523ebba3c05d327b2e39d18b6"
server_key_version: "encryptionkey@0"
I20260620 19:11:49.051870 20543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:11:49.052657 20557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:49.052814 20543 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:11:49.052915 20543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-0/wal
uuid: "fa2daecd7d9a4f4cb7e9fb9bc6ff43f9"
format_stamp: "Formatted at 2026-06-20 19:11:49 on dist-test-slave-wd30"
server_key: "d89173f074daf4a740f4be08aaec6ad0"
server_key_iv: "a343d03523ebba3c05d327b2e39d18b6"
server_key_version: "encryptionkey@0"
I20260620 19:11:49.053004 20543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:49.072700 20543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:49.073045 20543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:49.073200 20543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:49.073452 20543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:49.073815 20543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:49.073850 20543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:49.073873 20543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:49.073920 20543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:49.080037 20543 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:43649
I20260620 19:11:49.080075 20670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:43649 every 8 connection(s)
I20260620 19:11:49.080404 20543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-0/data/info.pb
I20260620 19:11:49.085530 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:49.081126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.1:55615 (local address 127.0.203.62:44961)
0620 19:11:49.081210 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:49.081212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:49.081473 (+ 261us) server_negotiation.cc:407] Connection header received
0620 19:11:49.081971 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:49.081972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:49.081984 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:49.082045 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:49.082719 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.083279 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.084236 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.084419 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.084469 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:49.085305 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:49.085306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:49.085306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:49.085311 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:49.085323 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:49.085327 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:49.085367 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:49.085449 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 19:11:49.085473 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:11:49.085774 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:49.081103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:44961 (local address 127.0.203.1:55615)
0620 19:11:49.081325 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:49.081368 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:49.081884 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:49.082114 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:49.082118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:49.082126 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:49.082644 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:49.082648 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.083359 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.083360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:49.084157 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:49.084159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.084517 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.084519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:49.084649 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:49.085172 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:11:49.085173 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:49.085201 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:11:49.085224 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:49.085380 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:49.085384 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:49.085387 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:11:49.085437 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 19:11:49.085518 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":110,"threads_started":1}
I20260620 19:11:49.086251 20671 heartbeater.cc:347] Connected to a master server at 127.0.203.62:44961
I20260620 19:11:49.086340 20671 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:49.086481 20671 heartbeater.cc:511] Master 127.0.203.62:44961 requested a full tablet report, sending...
I20260620 19:11:49.086982 20483 ts_manager.cc:194] Registered new tserver with Master: fa2daecd7d9a4f4cb7e9fb9bc6ff43f9 (127.0.203.1:43649)
I20260620 19:11:49.087641 20483 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.1:55615
I20260620 19:11:49.088027 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 20543
I20260620 19:11:49.088145 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-0/wal/instance
I20260620 19:11:49.088348 812 external_mini_cluster.cc:1428] Setting key f2bb59da5ef0de8d6ade942280c640fa
I20260620 19:11:49.089206 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.2:0
--local_ip_for_outbound_sockets=127.0.203.2
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44961
--builtin_ntp_servers=127.0.203.20:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:11:49.172385 20674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:49.172636 20674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:49.172675 20674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:49.173928 20674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:49.173990 20674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:49.174034 20674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.2
I20260620 19:11:49.175446 20674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.2
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.2
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:49.175706 20674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:49.176008 20674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:49.180894 20679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:49.180907 20680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:49.180907 20682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:49.181213 20674 server_base.cc:1034] running on GCE node
I20260620 19:11:49.181408 20674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:49.181627 20674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:49.182802 20674 hybrid_clock.cc:648] HybridClock initialized: now 1781982709182786 us; error 41 us; skew 500 ppm
I20260620 19:11:49.184072 20674 webserver.cc:460] Webserver started at http://127.0.203.2:39063/ using document root <none> and password file <none>
I20260620 19:11:49.184316 20674 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:49.184388 20674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:49.184567 20674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:49.185635 20674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-1/data/instance:
uuid: "3dbaa27e4f934b89b0f24e6e0dc90192"
format_stamp: "Formatted at 2026-06-20 19:11:49 on dist-test-slave-wd30"
server_key: "37e33868083720e37d989e1681642c45"
server_key_iv: "63fd72b8571caa6ce6831dbbdbf7841f"
server_key_version: "encryptionkey@0"
I20260620 19:11:49.186008 20674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-1/wal/instance:
uuid: "3dbaa27e4f934b89b0f24e6e0dc90192"
format_stamp: "Formatted at 2026-06-20 19:11:49 on dist-test-slave-wd30"
server_key: "37e33868083720e37d989e1681642c45"
server_key_iv: "63fd72b8571caa6ce6831dbbdbf7841f"
server_key_version: "encryptionkey@0"
I20260620 19:11:49.187361 20674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:11:49.188084 20688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:49.188249 20674 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:11:49.188351 20674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-1/wal
uuid: "3dbaa27e4f934b89b0f24e6e0dc90192"
format_stamp: "Formatted at 2026-06-20 19:11:49 on dist-test-slave-wd30"
server_key: "37e33868083720e37d989e1681642c45"
server_key_iv: "63fd72b8571caa6ce6831dbbdbf7841f"
server_key_version: "encryptionkey@0"
I20260620 19:11:49.188433 20674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:49.202992 20674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:49.203343 20674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:49.203513 20674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:49.203756 20674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:49.204088 20674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:49.204144 20674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:49.204200 20674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:49.204228 20674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:49.210160 20674 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:33365
I20260620 19:11:49.210279 20801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:33365 every 8 connection(s)
I20260620 19:11:49.210621 20674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-1/data/info.pb
I20260620 19:11:49.214066 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 20674
I20260620 19:11:49.214191 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-1/wal/instance
I20260620 19:11:49.214402 812 external_mini_cluster.cc:1428] Setting key 1dc91242221d0ac957b2b43cab4e066f
I20260620 19:11:49.215688 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.3:0
--local_ip_for_outbound_sockets=127.0.203.3
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44961
--builtin_ntp_servers=127.0.203.20:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:11:49.217118 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:49.211533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.2:57047 (local address 127.0.203.62:44961)
0620 19:11:49.211609 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:49.211611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:49.211841 (+ 230us) server_negotiation.cc:407] Connection header received
0620 19:11:49.212368 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:49.212369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:49.212380 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:49.212440 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:49.213180 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.213740 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.215400 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.215631 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.215722 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:49.216808 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:49.216809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:49.216809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:49.216821 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:49.216836 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:49.216840 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:49.216901 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:49.217001 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 19:11:49.217035 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:11:49.217377 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:49.211496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:44961 (local address 127.0.203.2:57047)
0620 19:11:49.211701 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:49.211744 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:49.212271 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:49.212511 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:49.212514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:49.212527 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:49.213060 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:49.213064 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.213887 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.213889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:49.215224 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:49.215228 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.215734 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.215735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:49.215912 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:49.216632 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:11:49.216632 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:49.216665 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:11:49.216691 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:49.216901 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:49.216909 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:49.216913 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:11:49.217004 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 19:11:49.217131 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":123,"threads_started":1}
I20260620 19:11:49.217952 20802 heartbeater.cc:347] Connected to a master server at 127.0.203.62:44961
I20260620 19:11:49.218071 20802 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:49.218307 20802 heartbeater.cc:511] Master 127.0.203.62:44961 requested a full tablet report, sending...
I20260620 19:11:49.218698 20483 ts_manager.cc:194] Registered new tserver with Master: 3dbaa27e4f934b89b0f24e6e0dc90192 (127.0.203.2:33365)
I20260620 19:11:49.219105 20483 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.2:57047
W20260620 19:11:49.297097 20805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:49.297353 20805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:49.297389 20805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:49.298590 20805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:49.298671 20805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:49.298712 20805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.3
I20260620 19:11:49.300103 20805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.3
--webserver_port=0
--tserver_master_addrs=127.0.203.62:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.3
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:49.300395 20805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:49.300727 20805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:49.305835 20811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:49.305924 20805 server_base.cc:1034] running on GCE node
W20260620 19:11:49.305826 20810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:49.305830 20813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:49.306285 20805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:49.306540 20805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:49.307698 20805 hybrid_clock.cc:648] HybridClock initialized: now 1781982709307688 us; error 40 us; skew 500 ppm
I20260620 19:11:49.309109 20805 webserver.cc:460] Webserver started at http://127.0.203.3:36959/ using document root <none> and password file <none>
I20260620 19:11:49.309374 20805 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:49.309445 20805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:49.309587 20805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:49.310688 20805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-2/data/instance:
uuid: "c8f690c3bdcb4bb992b311f200715a35"
format_stamp: "Formatted at 2026-06-20 19:11:49 on dist-test-slave-wd30"
server_key: "c311997cadc5ce475976a052ca3eabf1"
server_key_iv: "6e370d1374366fa0b9d2e4abdb7461d8"
server_key_version: "encryptionkey@0"
I20260620 19:11:49.311066 20805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-2/wal/instance:
uuid: "c8f690c3bdcb4bb992b311f200715a35"
format_stamp: "Formatted at 2026-06-20 19:11:49 on dist-test-slave-wd30"
server_key: "c311997cadc5ce475976a052ca3eabf1"
server_key_iv: "6e370d1374366fa0b9d2e4abdb7461d8"
server_key_version: "encryptionkey@0"
I20260620 19:11:49.312378 20805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:11:49.313103 20819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:49.313268 20805 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:11:49.313369 20805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-2/wal
uuid: "c8f690c3bdcb4bb992b311f200715a35"
format_stamp: "Formatted at 2026-06-20 19:11:49 on dist-test-slave-wd30"
server_key: "c311997cadc5ce475976a052ca3eabf1"
server_key_iv: "6e370d1374366fa0b9d2e4abdb7461d8"
server_key_version: "encryptionkey@0"
I20260620 19:11:49.313447 20805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:49.325243 20805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:49.325557 20805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:49.325707 20805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:49.325933 20805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:49.326272 20805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:49.326328 20805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:49.326370 20805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:49.326398 20805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:49.332224 20805 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:37489
I20260620 19:11:49.332310 20932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:37489 every 8 connection(s)
I20260620 19:11:49.332620 20805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-2/data/info.pb
I20260620 19:11:49.337980 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:49.333320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:46093 (local address 127.0.203.62:44961)
0620 19:11:49.333364 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:49.333367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:49.333605 (+ 238us) server_negotiation.cc:407] Connection header received
0620 19:11:49.334147 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:49.334148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:49.334162 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:49.334213 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:49.334900 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.335507 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.336515 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.336721 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.336798 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:49.337658 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:49.337659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:49.337660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:49.337665 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:49.337681 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:49.337685 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:49.337754 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:49.337848 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 19:11:49.337880 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:11:49.338199 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:49.333306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:44961 (local address 127.0.203.3:46093)
0620 19:11:49.333474 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:49.333507 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:49.334029 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:49.334281 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:49.334285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:49.334301 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:49.334800 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:49.334805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.335605 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.335606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:49.336396 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:49.336398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.336808 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.336809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:49.336941 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:49.337498 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:11:49.337499 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:49.337528 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:11:49.337551 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:49.337755 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:49.337761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:49.337765 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:11:49.337851 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 19:11:49.337960 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":68,"threads_started":1}
I20260620 19:11:49.338726 20933 heartbeater.cc:347] Connected to a master server at 127.0.203.62:44961
I20260620 19:11:49.338806 20933 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:49.339011 20933 heartbeater.cc:511] Master 127.0.203.62:44961 requested a full tablet report, sending...
I20260620 19:11:49.339376 20483 ts_manager.cc:194] Registered new tserver with Master: c8f690c3bdcb4bb992b311f200715a35 (127.0.203.3:37489)
I20260620 19:11:49.339705 20483 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.3:46093
I20260620 19:11:49.341588 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 20805
I20260620 19:11:49.341670 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-2/wal/instance
I20260620 19:11:49.341876 812 external_mini_cluster.cc:1428] Setting key e93bb35687efe46d735c8a78e01481db
I20260620 19:11:49.342723 812 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:11:49.346985 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:49.344226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46862 (local address 127.0.203.62:44961)
0620 19:11:49.344290 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:49.344292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:49.344413 (+ 121us) server_negotiation.cc:407] Connection header received
0620 19:11:49.344450 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:49.344451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:49.344462 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:49.344511 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:49.344672 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.345141 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.345883 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.346071 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.346137 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:49.346727 (+ 590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:49.346728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:49.346729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:49.346733 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:49.346745 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:49.346749 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:49.346792 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:49.346839 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 19:11:49.346863 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:11:49.348666 20483 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:11:49.350188 20483 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:11:49.356824 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:49.352772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.0.203.1:43649)
0620 19:11:49.352911 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:49.352916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:49.353368 (+ 452us) server_negotiation.cc:407] Connection header received
0620 19:11:49.353412 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:49.353415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:49.353430 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:49.353504 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:49.353628 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.354607 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.355706 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.356383 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.356475 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:49.356568 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:49.356697 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 19:11:49.356734 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20260620 19:11:49.356926 20941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:49.352405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:33365 (local address 127.0.0.1:35996)
0620 19:11:49.352580 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:49.352621 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:49.352685 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:49.352925 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:49.352928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:49.352930 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:49.352995 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:49.352996 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.354398 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.354399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:49.355649 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:49.355651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.356633 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.356635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:49.356705 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:49.356709 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:11:49.356740 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 19:11:49.356788 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":27,"thread_start_us":111,"threads_started":1}
I20260620 19:11:49.356813 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:49.352416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:43649 (local address 127.0.0.1:46904)
0620 19:11:49.353248 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:49.353258 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:49.353308 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:49.353520 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:49.353520 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:49.353530 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:49.353565 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:49.353567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.354694 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.354696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:49.355612 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:49.355614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.356478 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.356479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:49.356563 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:49.356578 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 19:11:49.356679 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 19:11:49.356736 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":165,"threads_started":1}
I20260620 19:11:49.357004 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:49.352671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.0.203.3:37489)
0620 19:11:49.352948 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:49.352951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:49.352960 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:11:49.353019 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:49.353021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:49.353032 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:49.353146 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:49.353319 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.354152 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.355745 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.356489 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.356576 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:49.356668 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:49.356876 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 19:11:49.356918 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":77,"threads_started":1}
I20260620 19:11:49.357116 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:49.352592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35996 (local address 127.0.203.2:33365)
0620 19:11:49.352763 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:49.352769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:49.352780 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:11:49.352822 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:49.352824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:49.352842 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:49.352924 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:49.353157 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.354318 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.355745 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.356489 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.356859 (+ 370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:49.356945 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:49.356968 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 19:11:49.357006 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260620 19:11:49.357178 20942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:49.352409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.3:37489 (local address 127.0.0.1:49118)
0620 19:11:49.352784 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:49.352793 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:49.352828 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:49.353191 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:49.353191 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:49.353193 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:49.353236 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:49.353237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.354243 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.354244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:49.355645 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:49.355648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.356697 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.356698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:49.356753 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:49.356756 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:11:49.356799 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 19:11:49.356836 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":3,"thread_start_us":67,"threads_started":1}
I20260620 19:11:49.357897 20867 tablet_service.cc:1467] Processing CreateTablet for tablet deafdaa1190441778356709d48ad5d61 (DEFAULT_TABLE table=test-table [id=d8dbae18832d4add92fdcc2042a95c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:49.357852 20736 tablet_service.cc:1467] Processing CreateTablet for tablet deafdaa1190441778356709d48ad5d61 (DEFAULT_TABLE table=test-table [id=d8dbae18832d4add92fdcc2042a95c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:49.358266 20867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deafdaa1190441778356709d48ad5d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:49.358271 20736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deafdaa1190441778356709d48ad5d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:49.358381 20605 tablet_service.cc:1467] Processing CreateTablet for tablet deafdaa1190441778356709d48ad5d61 (DEFAULT_TABLE table=test-table [id=d8dbae18832d4add92fdcc2042a95c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:49.358698 20605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deafdaa1190441778356709d48ad5d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:49.361119 20947 tablet_bootstrap.cc:492] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35: Bootstrap starting.
I20260620 19:11:49.361688 20948 tablet_bootstrap.cc:492] T deafdaa1190441778356709d48ad5d61 P 3dbaa27e4f934b89b0f24e6e0dc90192: Bootstrap starting.
I20260620 19:11:49.361744 20947 tablet_bootstrap.cc:654] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:49.362043 20949 tablet_bootstrap.cc:492] T deafdaa1190441778356709d48ad5d61 P fa2daecd7d9a4f4cb7e9fb9bc6ff43f9: Bootstrap starting.
I20260620 19:11:49.362094 20947 log.cc:824] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:49.362455 20948 tablet_bootstrap.cc:654] T deafdaa1190441778356709d48ad5d61 P 3dbaa27e4f934b89b0f24e6e0dc90192: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:49.362679 20949 tablet_bootstrap.cc:654] T deafdaa1190441778356709d48ad5d61 P fa2daecd7d9a4f4cb7e9fb9bc6ff43f9: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:49.362782 20948 log.cc:824] T deafdaa1190441778356709d48ad5d61 P 3dbaa27e4f934b89b0f24e6e0dc90192: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:49.362936 20947 tablet_bootstrap.cc:492] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35: No bootstrap required, opened a new log
I20260620 19:11:49.363006 20947 ts_tablet_manager.cc:1397] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:11:49.362999 20949 log.cc:824] T deafdaa1190441778356709d48ad5d61 P fa2daecd7d9a4f4cb7e9fb9bc6ff43f9: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:49.363574 20948 tablet_bootstrap.cc:492] T deafdaa1190441778356709d48ad5d61 P 3dbaa27e4f934b89b0f24e6e0dc90192: No bootstrap required, opened a new log
I20260620 19:11:49.363654 20948 ts_tablet_manager.cc:1397] T deafdaa1190441778356709d48ad5d61 P 3dbaa27e4f934b89b0f24e6e0dc90192: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:11:49.365724 20947 raft_consensus.cc:348] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbaa27e4f934b89b0f24e6e0dc90192" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33365 } } peers { permanent_uuid: "fa2daecd7d9a4f4cb7e9fb9bc6ff43f9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43649 } } peers { permanent_uuid: "c8f690c3bdcb4bb992b311f200715a35" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37489 } }
I20260620 19:11:49.365906 20947 raft_consensus.cc:374] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:49.365942 20947 raft_consensus.cc:729] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f690c3bdcb4bb992b311f200715a35, State: Initialized, Role: FOLLOWER
I20260620 19:11:49.366086 20947 consensus_queue.cc:260] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbaa27e4f934b89b0f24e6e0dc90192" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33365 } } peers { permanent_uuid: "fa2daecd7d9a4f4cb7e9fb9bc6ff43f9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43649 } } peers { permanent_uuid: "c8f690c3bdcb4bb992b311f200715a35" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37489 } }
I20260620 19:11:49.366211 20948 raft_consensus.cc:348] T deafdaa1190441778356709d48ad5d61 P 3dbaa27e4f934b89b0f24e6e0dc90192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbaa27e4f934b89b0f24e6e0dc90192" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33365 } } peers { permanent_uuid: "fa2daecd7d9a4f4cb7e9fb9bc6ff43f9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43649 } } peers { permanent_uuid: "c8f690c3bdcb4bb992b311f200715a35" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37489 } }
I20260620 19:11:49.366386 20948 raft_consensus.cc:374] T deafdaa1190441778356709d48ad5d61 P 3dbaa27e4f934b89b0f24e6e0dc90192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:49.366436 20948 raft_consensus.cc:729] T deafdaa1190441778356709d48ad5d61 P 3dbaa27e4f934b89b0f24e6e0dc90192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dbaa27e4f934b89b0f24e6e0dc90192, State: Initialized, Role: FOLLOWER
I20260620 19:11:49.366501 20947 ts_tablet_manager.cc:1428] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:11:49.366551 20948 consensus_queue.cc:260] T deafdaa1190441778356709d48ad5d61 P 3dbaa27e4f934b89b0f24e6e0dc90192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbaa27e4f934b89b0f24e6e0dc90192" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33365 } } peers { permanent_uuid: "fa2daecd7d9a4f4cb7e9fb9bc6ff43f9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43649 } } peers { permanent_uuid: "c8f690c3bdcb4bb992b311f200715a35" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37489 } }
I20260620 19:11:49.366587 20933 heartbeater.cc:503] Master 127.0.203.62:44961 was elected leader, sending a full tablet report...
I20260620 19:11:49.366852 20948 ts_tablet_manager.cc:1428] T deafdaa1190441778356709d48ad5d61 P 3dbaa27e4f934b89b0f24e6e0dc90192: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:11:49.366886 20802 heartbeater.cc:503] Master 127.0.203.62:44961 was elected leader, sending a full tablet report...
I20260620 19:11:49.367370 20949 tablet_bootstrap.cc:492] T deafdaa1190441778356709d48ad5d61 P fa2daecd7d9a4f4cb7e9fb9bc6ff43f9: No bootstrap required, opened a new log
I20260620 19:11:49.367450 20949 ts_tablet_manager.cc:1397] T deafdaa1190441778356709d48ad5d61 P fa2daecd7d9a4f4cb7e9fb9bc6ff43f9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 19:11:49.369534 20949 raft_consensus.cc:348] T deafdaa1190441778356709d48ad5d61 P fa2daecd7d9a4f4cb7e9fb9bc6ff43f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbaa27e4f934b89b0f24e6e0dc90192" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33365 } } peers { permanent_uuid: "fa2daecd7d9a4f4cb7e9fb9bc6ff43f9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43649 } } peers { permanent_uuid: "c8f690c3bdcb4bb992b311f200715a35" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37489 } }
I20260620 19:11:49.369658 20949 raft_consensus.cc:374] T deafdaa1190441778356709d48ad5d61 P fa2daecd7d9a4f4cb7e9fb9bc6ff43f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:49.369679 20949 raft_consensus.cc:729] T deafdaa1190441778356709d48ad5d61 P fa2daecd7d9a4f4cb7e9fb9bc6ff43f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa2daecd7d9a4f4cb7e9fb9bc6ff43f9, State: Initialized, Role: FOLLOWER
I20260620 19:11:49.369819 20949 consensus_queue.cc:260] T deafdaa1190441778356709d48ad5d61 P fa2daecd7d9a4f4cb7e9fb9bc6ff43f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbaa27e4f934b89b0f24e6e0dc90192" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33365 } } peers { permanent_uuid: "fa2daecd7d9a4f4cb7e9fb9bc6ff43f9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43649 } } peers { permanent_uuid: "c8f690c3bdcb4bb992b311f200715a35" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37489 } }
I20260620 19:11:49.370077 20949 ts_tablet_manager.cc:1428] T deafdaa1190441778356709d48ad5d61 P fa2daecd7d9a4f4cb7e9fb9bc6ff43f9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:11:49.370137 20671 heartbeater.cc:503] Master 127.0.203.62:44961 was elected leader, sending a full tablet report...
W20260620 19:11:49.461225 20803 tablet.cc:2369] T deafdaa1190441778356709d48ad5d61 P 3dbaa27e4f934b89b0f24e6e0dc90192: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:11:49.525847 20953 raft_consensus.cc:482] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:11:49.525985 20953 raft_consensus.cc:504] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbaa27e4f934b89b0f24e6e0dc90192" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33365 } } peers { permanent_uuid: "fa2daecd7d9a4f4cb7e9fb9bc6ff43f9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43649 } } peers { permanent_uuid: "c8f690c3bdcb4bb992b311f200715a35" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37489 } }
I20260620 19:11:49.526463 20953 leader_election.cc:288] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3dbaa27e4f934b89b0f24e6e0dc90192 (127.0.203.2:33365), fa2daecd7d9a4f4cb7e9fb9bc6ff43f9 (127.0.203.1:43649)
I20260620 19:11:49.529444 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:49.526813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:51771 (local address 127.0.203.1:43649)
0620 19:11:49.526856 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:49.526858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:49.526970 (+ 112us) server_negotiation.cc:407] Connection header received
0620 19:11:49.527035 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:49.527036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:49.527049 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:49.527102 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:49.527318 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.527775 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.528716 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.529076 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.529162 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:49.529172 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:49.529343 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 19:11:49.529381 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:11:49.529436 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:49.526697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:33365 (local address 127.0.203.3:60949)
0620 19:11:49.526769 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:49.526783 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:49.526853 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:49.527040 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:49.527041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:49.527043 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:49.527141 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:49.527143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.527837 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.527838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:49.528610 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:49.528612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.529164 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.529165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:49.529242 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:49.529245 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:11:49.529287 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 19:11:49.529352 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":2}
I20260620 19:11:49.529438 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:49.526844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.3:60949 (local address 127.0.203.2:33365)
0620 19:11:49.526891 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:49.526893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:49.526905 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:11:49.526942 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:49.526943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:49.526952 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:49.527010 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:49.527222 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.527746 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.528711 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.529075 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.529163 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:49.529174 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:49.529320 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 19:11:49.529364 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:11:49.529456 20956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:49.526708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:43649 (local address 127.0.203.3:51771)
0620 19:11:49.526887 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:49.526898 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:49.526962 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:49.527157 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:49.527158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:49.527160 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:49.527238 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:49.527240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.527883 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.527884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:49.528615 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:49.528616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.529179 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.529180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:49.529245 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:49.529248 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:11:49.529324 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:11:49.529361 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":14,"thread_start_us":98,"threads_started":1}
I20260620 19:11:49.529887 20625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deafdaa1190441778356709d48ad5d61" candidate_uuid: "c8f690c3bdcb4bb992b311f200715a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa2daecd7d9a4f4cb7e9fb9bc6ff43f9" is_pre_election: true
I20260620 19:11:49.529891 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deafdaa1190441778356709d48ad5d61" candidate_uuid: "c8f690c3bdcb4bb992b311f200715a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dbaa27e4f934b89b0f24e6e0dc90192" is_pre_election: true
I20260620 19:11:49.530058 20625 raft_consensus.cc:2454] T deafdaa1190441778356709d48ad5d61 P fa2daecd7d9a4f4cb7e9fb9bc6ff43f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f690c3bdcb4bb992b311f200715a35 in term 0.
I20260620 19:11:49.530066 20756 raft_consensus.cc:2454] T deafdaa1190441778356709d48ad5d61 P 3dbaa27e4f934b89b0f24e6e0dc90192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f690c3bdcb4bb992b311f200715a35 in term 0.
I20260620 19:11:49.530315 20821 leader_election.cc:302] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8f690c3bdcb4bb992b311f200715a35, fa2daecd7d9a4f4cb7e9fb9bc6ff43f9; no voters:
I20260620 19:11:49.530457 20953 raft_consensus.cc:2781] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:11:49.530522 20953 raft_consensus.cc:482] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:11:49.530545 20953 raft_consensus.cc:3037] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:49.531253 20953 raft_consensus.cc:504] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbaa27e4f934b89b0f24e6e0dc90192" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33365 } } peers { permanent_uuid: "fa2daecd7d9a4f4cb7e9fb9bc6ff43f9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43649 } } peers { permanent_uuid: "c8f690c3bdcb4bb992b311f200715a35" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37489 } }
I20260620 19:11:49.531400 20953 leader_election.cc:288] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [CANDIDATE]: Term 1 election: Requested vote from peers 3dbaa27e4f934b89b0f24e6e0dc90192 (127.0.203.2:33365), fa2daecd7d9a4f4cb7e9fb9bc6ff43f9 (127.0.203.1:43649)
I20260620 19:11:49.531560 20625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deafdaa1190441778356709d48ad5d61" candidate_uuid: "c8f690c3bdcb4bb992b311f200715a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa2daecd7d9a4f4cb7e9fb9bc6ff43f9"
I20260620 19:11:49.531585 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deafdaa1190441778356709d48ad5d61" candidate_uuid: "c8f690c3bdcb4bb992b311f200715a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dbaa27e4f934b89b0f24e6e0dc90192"
I20260620 19:11:49.531656 20625 raft_consensus.cc:3037] T deafdaa1190441778356709d48ad5d61 P fa2daecd7d9a4f4cb7e9fb9bc6ff43f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:49.531666 20756 raft_consensus.cc:3037] T deafdaa1190441778356709d48ad5d61 P 3dbaa27e4f934b89b0f24e6e0dc90192 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:49.532356 20625 raft_consensus.cc:2454] T deafdaa1190441778356709d48ad5d61 P fa2daecd7d9a4f4cb7e9fb9bc6ff43f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f690c3bdcb4bb992b311f200715a35 in term 1.
I20260620 19:11:49.532362 20756 raft_consensus.cc:2454] T deafdaa1190441778356709d48ad5d61 P 3dbaa27e4f934b89b0f24e6e0dc90192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f690c3bdcb4bb992b311f200715a35 in term 1.
I20260620 19:11:49.532562 20823 leader_election.cc:302] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dbaa27e4f934b89b0f24e6e0dc90192, c8f690c3bdcb4bb992b311f200715a35; no voters:
I20260620 19:11:49.532666 20953 raft_consensus.cc:2781] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:49.532825 20953 raft_consensus.cc:686] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [term 1 LEADER]: Becoming Leader. State: Replica: c8f690c3bdcb4bb992b311f200715a35, State: Running, Role: LEADER
I20260620 19:11:49.532959 20953 consensus_queue.cc:237] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbaa27e4f934b89b0f24e6e0dc90192" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33365 } } peers { permanent_uuid: "fa2daecd7d9a4f4cb7e9fb9bc6ff43f9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43649 } } peers { permanent_uuid: "c8f690c3bdcb4bb992b311f200715a35" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37489 } }
I20260620 19:11:49.533785 20483 catalog_manager.cc:5495] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8f690c3bdcb4bb992b311f200715a35 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "c8f690c3bdcb4bb992b311f200715a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbaa27e4f934b89b0f24e6e0dc90192" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa2daecd7d9a4f4cb7e9fb9bc6ff43f9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f690c3bdcb4bb992b311f200715a35" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 37489 } health_report { overall_health: HEALTHY } } }
W20260620 19:11:49.581177 20672 tablet.cc:2369] T deafdaa1190441778356709d48ad5d61 P fa2daecd7d9a4f4cb7e9fb9bc6ff43f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:11:49.582324 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:49.579732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49130 (local address 127.0.203.3:37489)
0620 19:11:49.579789 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:49.579791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:49.579799 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:11:49.579848 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:49.579848 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:49.579866 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:49.579930 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:11:49.580137 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.580621 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.581502 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:49.581699 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:49.581788 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:49.581961 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:11:49.582066 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:11:49.582153 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:49.582191 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 19:11:49.582209 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 19:11:49.583173 20934 tablet.cc:2369] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:11:49.584158 20756 raft_consensus.cc:1261] T deafdaa1190441778356709d48ad5d61 P 3dbaa27e4f934b89b0f24e6e0dc90192 [term 1 FOLLOWER]: Refusing update from remote peer c8f690c3bdcb4bb992b311f200715a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:11:49.584223 20625 raft_consensus.cc:1261] T deafdaa1190441778356709d48ad5d61 P fa2daecd7d9a4f4cb7e9fb9bc6ff43f9 [term 1 FOLLOWER]: Refusing update from remote peer c8f690c3bdcb4bb992b311f200715a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:11:49.584530 20957 consensus_queue.cc:1035] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dbaa27e4f934b89b0f24e6e0dc90192" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 33365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:11:49.584659 20953 consensus_queue.cc:1035] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa2daecd7d9a4f4cb7e9fb9bc6ff43f9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 43649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:11:49.587033 20960 mvcc.cc:205] Tried to move back new op lower bound from 7299001178454016000 to 7299001178247790592. Current Snapshot: MvccSnapshot[applied={T|T < 7299001178454016000}]
I20260620 19:11:49.587244 20961 mvcc.cc:205] Tried to move back new op lower bound from 7299001178454016000 to 7299001178247790592. Current Snapshot: MvccSnapshot[applied={T|T < 7299001178454016000}]
I20260620 19:11:49.589820 20483 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:11:49.589916 20483 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:11:49.590641 20483 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6506212edf8a4419a9723c221d3937d4: Sending DeleteTablet for 3 replicas of tablet deafdaa1190441778356709d48ad5d61
I20260620 19:11:49.590962 20867 tablet_service.cc:1514] Processing DeleteTablet for tablet deafdaa1190441778356709d48ad5d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:49 UTC) from {username='slave'} at 127.0.0.1:49118
I20260620 19:11:49.590982 20605 tablet_service.cc:1514] Processing DeleteTablet for tablet deafdaa1190441778356709d48ad5d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:49 UTC) from {username='slave'} at 127.0.0.1:46904
I20260620 19:11:49.591027 20736 tablet_service.cc:1514] Processing DeleteTablet for tablet deafdaa1190441778356709d48ad5d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:49 UTC) from {username='slave'} at 127.0.0.1:35996
I20260620 19:11:49.591219 20970 tablet_replica.cc:330] T deafdaa1190441778356709d48ad5d61 P fa2daecd7d9a4f4cb7e9fb9bc6ff43f9: stopping tablet replica
I20260620 19:11:49.591315 20970 raft_consensus.cc:2229] T deafdaa1190441778356709d48ad5d61 P fa2daecd7d9a4f4cb7e9fb9bc6ff43f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:11:49.591334 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 20543
I20260620 19:11:49.592013 20971 tablet_replica.cc:330] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35: stopping tablet replica
I20260620 19:11:49.592104 20971 raft_consensus.cc:2229] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:11:49.592211 20971 raft_consensus.cc:2258] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:11:49.595497 20971 ts_tablet_manager.cc:1905] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:11:49.596529 20468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.1:43649 (error 108)
I20260620 19:11:49.596776 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 20674
I20260620 19:11:49.597280 20971 ts_tablet_manager.cc:1918] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:11:49.597353 20971 log.cc:1196] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35: Deleting WAL directory at /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982468536772-812-0/minicluster-data/ts-2/wal/wals/deafdaa1190441778356709d48ad5d61
I20260620 19:11:49.597620 20971 ts_tablet_manager.cc:1939] T deafdaa1190441778356709d48ad5d61 P c8f690c3bdcb4bb992b311f200715a35: Deleting consensus metadata
I20260620 19:11:49.598023 20469 catalog_manager.cc:4841] TS c8f690c3bdcb4bb992b311f200715a35 (127.0.203.3:37489): tablet deafdaa1190441778356709d48ad5d61 (table test-table [id=d8dbae18832d4add92fdcc2042a95c45]) successfully deleted
I20260620 19:11:49.603212 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 20805
I20260620 19:11:49.603359 20942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:49.603201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.1:43649 (local address 127.0.0.1:46920)
0620 19:11:49.603273 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:49.603325 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.1:43649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 19:11:49.603443 20942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:49.603299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.2:33365 (local address 127.0.0.1:36002)
0620 19:11:49.603419 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:49.603434 (+ 15us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.2:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260620 19:11:49.605142 20470 catalog_manager.cc:4584] TS 3dbaa27e4f934b89b0f24e6e0dc90192 (127.0.203.2:33365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet deafdaa1190441778356709d48ad5d61: Network error: Client connection negotiation failed: client connection to 127.0.203.2:33365: connect: Connection refused (error 111)
I20260620 19:11:49.608604 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 20452
2026-06-20T19:11:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (865 ms)
[----------] 6 tests from EncryptionPolicyTest (8761 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:11:49 dist-test-slave-wd30 krb5kdc[20976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:11:49 dist-test-slave-wd30 krb5kdc[20976](info): set up 2 sockets
Jun 20 19:11:49 dist-test-slave-wd30 krb5kdc[20976](info): commencing operation
krb5kdc: starting...
W20260620 19:11:50.647610 812 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:11:50 dist-test-slave-wd30 krb5kdc[20976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:11:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:11:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:11:50.766237 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:50.845912 20989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:50.846164 20989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:50.846205 20989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:50.847379 20989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:11:50.847450 20989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:50.847476 20989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:50.847501 20989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:11:50.847532 20989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:11:50.848981 20989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:50.849246 20989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:50.849470 20989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:50.854290 20997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:50.854290 20994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:50.854319 20989 server_base.cc:1034] running on GCE node
W20260620 19:11:50.854324 20995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:50.854815 20989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:50.855152 20989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:50.856329 20989 hybrid_clock.cc:648] HybridClock initialized: now 1781982710856317 us; error 44 us; skew 500 ppm
Jun 20 19:11:50 dist-test-slave-wd30 krb5kdc[20976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:50.858770 20989 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:11:50.859306 20989 webserver.cc:460] Webserver started at http://127.0.0.1:38809/ using document root <none> and password file <none>
I20260620 19:11:50.859560 20989 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:50.859628 20989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:50.859763 20989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:50.860824 20989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "8748b899c074485caf7f86f1969aa3d7"
format_stamp: "Formatted at 2026-06-20 19:11:50 on dist-test-slave-wd30"
server_key: "c36710a2d9d280d2ed17cb48531a99a8"
server_key_iv: "1552662fdf3b595d0b3c4e9592d0d8da"
server_key_version: "encryptionkey@0"
I20260620 19:11:50.861192 20989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "8748b899c074485caf7f86f1969aa3d7"
format_stamp: "Formatted at 2026-06-20 19:11:50 on dist-test-slave-wd30"
server_key: "c36710a2d9d280d2ed17cb48531a99a8"
server_key_iv: "1552662fdf3b595d0b3c4e9592d0d8da"
server_key_version: "encryptionkey@0"
I20260620 19:11:50.862516 20989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:11:50.863317 21004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:50.863510 20989 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:11:50.863611 20989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "8748b899c074485caf7f86f1969aa3d7"
format_stamp: "Formatted at 2026-06-20 19:11:50 on dist-test-slave-wd30"
server_key: "c36710a2d9d280d2ed17cb48531a99a8"
server_key_iv: "1552662fdf3b595d0b3c4e9592d0d8da"
server_key_version: "encryptionkey@0"
I20260620 19:11:50.863696 20989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:50.883937 20989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:50.886085 20989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:50.886258 20989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:50.890403 20989 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43305
I20260620 19:11:50.890439 21056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43305 every 8 connection(s)
I20260620 19:11:50.890833 20989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:11:50.891065 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 20989
I20260620 19:11:50.891211 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:11:50.891435 812 external_mini_cluster.cc:1428] Setting key e94d3a88f3f8aaf8c73de1627930b382
I20260620 19:11:50.891925 21057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:50.895292 21057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7: Bootstrap starting.
I20260620 19:11:50.895969 21057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:50.896278 21057 log.cc:824] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:50.896981 21057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7: No bootstrap required, opened a new log
Jun 20 19:11:50 dist-test-slave-wd30 krb5kdc[20976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:11:50.899502 21057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8748b899c074485caf7f86f1969aa3d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43305 } }
I20260620 19:11:50.899664 21057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:50.899729 21057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8748b899c074485caf7f86f1969aa3d7, State: Initialized, Role: FOLLOWER
I20260620 19:11:50.899909 21057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8748b899c074485caf7f86f1969aa3d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43305 } }
I20260620 19:11:50.900012 21057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:11:50.900055 21057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:11:50.900146 21057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:50.900729 21057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8748b899c074485caf7f86f1969aa3d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43305 } }
I20260620 19:11:50.900817 21057 leader_election.cc:302] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8748b899c074485caf7f86f1969aa3d7; no voters:
I20260620 19:11:50.901012 21057 leader_election.cc:288] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:11:50.901068 21062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:50.901265 21062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7 [term 1 LEADER]: Becoming Leader. State: Replica: 8748b899c074485caf7f86f1969aa3d7, State: Running, Role: LEADER
I20260620 19:11:50.901438 21062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8748b899c074485caf7f86f1969aa3d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43305 } }
I20260620 19:11:50.901326 21057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:11:50.901846 21063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8748b899c074485caf7f86f1969aa3d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8748b899c074485caf7f86f1969aa3d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43305 } } }
I20260620 19:11:50.901849 21064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8748b899c074485caf7f86f1969aa3d7. Latest consensus state: current_term: 1 leader_uuid: "8748b899c074485caf7f86f1969aa3d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8748b899c074485caf7f86f1969aa3d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43305 } } }
I20260620 19:11:50.901957 21063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7 [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:50.901965 21064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7 [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:50.902796 21067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:11:50.903327 21067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:11:50.903519 21060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:50.892011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54293 (local address 127.0.0.1:43305)
0620 19:11:50.892215 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:50.892224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:50.892255 (+ 31us) server_negotiation.cc:407] Connection header received
0620 19:11:50.892789 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:50.892798 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:50.892861 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:50.893016 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:50.893622 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:50.894970 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:50.895837 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:50.896479 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:50.896595 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:50.899263 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:50.899267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:50.899267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:50.899278 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:50.901117 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:50.901666 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:50.901669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:50.901669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:50.901778 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:50.902034 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:50.902035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:50.902036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:50.902407 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:50.902555 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:50.903105 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 19:11:50.903257 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":95,"threads_started":1}
I20260620 19:11:50.904600 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:11:50.904938 21067 catalog_manager.cc:1347] Generated new cluster ID: 68076f7b2fb44beda56813069d7a74dd
I20260620 19:11:50.904985 21067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:11:50.910997 21067 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:11:50.911789 21067 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:11:50.913378 21060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:50.906120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32919 (local address 127.0.0.1:43305)
0620 19:11:50.906152 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:50.906154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:50.906246 (+ 92us) server_negotiation.cc:407] Connection header received
0620 19:11:50.906303 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:50.906304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:50.906312 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:50.906372 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:50.907015 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:50.907779 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:50.908856 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:50.909175 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:50.909240 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:50.910667 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:50.910669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:50.910669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:50.910676 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:50.912204 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:50.912589 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:50.912590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:50.912590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:50.912651 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:50.912880 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:50.912881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:50.912881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:50.912989 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:50.913098 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:50.913221 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 19:11:50.913282 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:11:50.914467 21067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8748b899c074485caf7f86f1969aa3d7: Generated new TSK 0
I20260620 19:11:50.914618 21067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:11:50.917936 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 20989
W20260620 19:11:50.922068 20974 connection.cc:535] client connection to 127.0.0.1:43305 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43305): Connection reset by peer (error 104)
2026-06-20T19:11:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:11:50 dist-test-slave-wd30 krb5kdc[21088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:11:50 dist-test-slave-wd30 krb5kdc[21088](info): set up 2 sockets
Jun 20 19:11:50 dist-test-slave-wd30 krb5kdc[21088](info): commencing operation
krb5kdc: starting...
W20260620 19:11:51.955341 812 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:11:52 dist-test-slave-wd30 krb5kdc[21088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:11:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:11:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:11:52.077601 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42891
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:52.161831 21101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:52.162038 21101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:52.162061 21101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:52.163296 21101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:11:52.163338 21101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:52.163353 21101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:52.163367 21101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:11:52.163406 21101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:11:52.164876 21101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42891
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:52.165127 21101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:52.165365 21101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:52.170403 21106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:52.170477 21107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:52.170415 21109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:52.170477 21101 server_base.cc:1034] running on GCE node
I20260620 19:11:52.170979 21101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:52.171327 21101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:52.172509 21101 hybrid_clock.cc:648] HybridClock initialized: now 1781982712172478 us; error 46 us; skew 500 ppm
Jun 20 19:11:52 dist-test-slave-wd30 krb5kdc[21088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:52.175324 21101 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:11:52.175827 21101 webserver.cc:460] Webserver started at http://127.0.0.1:33197/ using document root <none> and password file <none>
I20260620 19:11:52.176071 21101 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:52.176139 21101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:52.176294 21101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:52.177386 21101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "20e81206760a49fdb288c6065e2149ab"
format_stamp: "Formatted at 2026-06-20 19:11:52 on dist-test-slave-wd30"
server_key: "f22750abce88d66fec6486b6715b1b94"
server_key_iv: "61aeb52f00adab3f44092b0391a7d6cd"
server_key_version: "encryptionkey@0"
I20260620 19:11:52.177754 21101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "20e81206760a49fdb288c6065e2149ab"
format_stamp: "Formatted at 2026-06-20 19:11:52 on dist-test-slave-wd30"
server_key: "f22750abce88d66fec6486b6715b1b94"
server_key_iv: "61aeb52f00adab3f44092b0391a7d6cd"
server_key_version: "encryptionkey@0"
I20260620 19:11:52.179186 21101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:11:52.180007 21116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:52.180255 21101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:11:52.180358 21101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "20e81206760a49fdb288c6065e2149ab"
format_stamp: "Formatted at 2026-06-20 19:11:52 on dist-test-slave-wd30"
server_key: "f22750abce88d66fec6486b6715b1b94"
server_key_iv: "61aeb52f00adab3f44092b0391a7d6cd"
server_key_version: "encryptionkey@0"
I20260620 19:11:52.180449 21101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:52.190238 21101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:52.192456 21101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:52.192624 21101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:52.196820 21101 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42891
I20260620 19:11:52.196848 21168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42891 every 8 connection(s)
I20260620 19:11:52.197222 21101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:11:52.198102 21169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:52.200507 21169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab: Bootstrap starting.
I20260620 19:11:52.201128 21169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:52.201447 21169 log.cc:824] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:52.202181 21169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab: No bootstrap required, opened a new log
I20260620 19:11:52.202845 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 21101
I20260620 19:11:52.202930 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:11:52.203173 812 external_mini_cluster.cc:1428] Setting key d80d7a81e4a2fc45c64eac9c5b7131be
I20260620 19:11:52.204478 21169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e81206760a49fdb288c6065e2149ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42891 } }
I20260620 19:11:52.204663 21169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:52.204703 21169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e81206760a49fdb288c6065e2149ab, State: Initialized, Role: FOLLOWER
I20260620 19:11:52.204885 21169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e81206760a49fdb288c6065e2149ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42891 } }
I20260620 19:11:52.204975 21169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:11:52.205008 21169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:11:52.205067 21169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:52.205821 21169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e81206760a49fdb288c6065e2149ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42891 } }
I20260620 19:11:52.205929 21169 leader_election.cc:302] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20e81206760a49fdb288c6065e2149ab; no voters:
I20260620 19:11:52.206075 21169 leader_election.cc:288] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:11:52.206130 21174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:52.206311 21169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:11:52.206341 21174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab [term 1 LEADER]: Becoming Leader. State: Replica: 20e81206760a49fdb288c6065e2149ab, State: Running, Role: LEADER
I20260620 19:11:52.206535 21174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e81206760a49fdb288c6065e2149ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42891 } }
I20260620 19:11:52.206884 21175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20e81206760a49fdb288c6065e2149ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e81206760a49fdb288c6065e2149ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42891 } } }
I20260620 19:11:52.206959 21175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:52.207183 21176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20e81206760a49fdb288c6065e2149ab. Latest consensus state: current_term: 1 leader_uuid: "20e81206760a49fdb288c6065e2149ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e81206760a49fdb288c6065e2149ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42891 } } }
I20260620 19:11:52.207302 21176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:52.207657 21181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:11:52.208141 21181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:11:52.209940 21173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:52.203761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48307 (local address 127.0.0.1:42891)
0620 19:11:52.203963 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:52.203974 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:52.204021 (+ 47us) server_negotiation.cc:407] Connection header received
0620 19:11:52.204441 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:52.204449 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:52.204513 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:52.204667 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:52.204916 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:52.206626 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:52.207776 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:52.208360 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:52.208470 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:52.209230 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:52.209233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:52.209234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:52.209420 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:52.209451 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:52.209461 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:52.209528 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:52.209681 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 19:11:52.209715 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260620 19:11:52.210069 21181 catalog_manager.cc:1347] Generated new cluster ID: 72096ac587db45d4b2c478d4216e5ebd
I20260620 19:11:52.210107 21181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:11:52.210987 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:11:52.216424 21173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:52.212481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57655 (local address 127.0.0.1:42891)
0620 19:11:52.212545 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:52.212547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:52.212777 (+ 230us) server_negotiation.cc:407] Connection header received
0620 19:11:52.212808 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:52.212809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:52.212821 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:52.212890 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:52.213026 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:52.213816 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:52.214880 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:52.215306 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:52.215388 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:52.216147 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:52.216149 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:52.216149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:52.216154 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:52.216173 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:52.216179 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:52.216245 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:52.216320 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 19:11:52.216334 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:11:52.229028 21181 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:11:52.229590 21181 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:11:52.236874 21181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20e81206760a49fdb288c6065e2149ab: Generated new TSK 0
I20260620 19:11:52.237090 21181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:11:52.242111 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 21101
W20260620 19:11:52.246181 21086 connection.cc:535] client connection to 127.0.0.1:42891 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42891): Connection reset by peer (error 104)
2026-06-20T19:11:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:11:52 dist-test-slave-wd30 krb5kdc[21200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:11:52 dist-test-slave-wd30 krb5kdc[21200](info): set up 2 sockets
Jun 20 19:11:52 dist-test-slave-wd30 krb5kdc[21200](info): commencing operation
krb5kdc: starting...
W20260620 19:11:53.275074 812 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:11:53 dist-test-slave-wd30 krb5kdc[21200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:11:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:11:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:11:53.394079 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:53.476130 21213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:53.476358 21213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:53.476383 21213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:53.477595 21213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:11:53.477638 21213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:53.477653 21213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:53.477666 21213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:11:53.477728 21213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:11:53.479429 21213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40857
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:53.479722 21213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:53.479962 21213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:53.484807 21218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:53.484944 21213 server_base.cc:1034] running on GCE node
W20260620 19:11:53.484826 21219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:53.484843 21221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:53.485381 21213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:53.485719 21213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:53.486907 21213 hybrid_clock.cc:648] HybridClock initialized: now 1781982713486890 us; error 43 us; skew 500 ppm
Jun 20 19:11:53 dist-test-slave-wd30 krb5kdc[21200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:53.489542 21213 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:11:53.489964 21213 webserver.cc:460] Webserver started at http://127.0.0.1:39289/ using document root <none> and password file <none>
I20260620 19:11:53.490149 21213 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:53.490188 21213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:53.490280 21213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:53.491421 21213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "dfafce4c84dc4c65b14106ac45456505"
format_stamp: "Formatted at 2026-06-20 19:11:53 on dist-test-slave-wd30"
server_key: "f140d9309a3beab629052cca6482185f"
server_key_iv: "5b2c5461c02745c9009801dc930cb08d"
server_key_version: "encryptionkey@0"
I20260620 19:11:53.491729 21213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "dfafce4c84dc4c65b14106ac45456505"
format_stamp: "Formatted at 2026-06-20 19:11:53 on dist-test-slave-wd30"
server_key: "f140d9309a3beab629052cca6482185f"
server_key_iv: "5b2c5461c02745c9009801dc930cb08d"
server_key_version: "encryptionkey@0"
I20260620 19:11:53.493090 21213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:11:53.493841 21228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:53.494102 21213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:11:53.494189 21213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "dfafce4c84dc4c65b14106ac45456505"
format_stamp: "Formatted at 2026-06-20 19:11:53 on dist-test-slave-wd30"
server_key: "f140d9309a3beab629052cca6482185f"
server_key_iv: "5b2c5461c02745c9009801dc930cb08d"
server_key_version: "encryptionkey@0"
I20260620 19:11:53.494248 21213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:53.497570 21213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:53.499972 21213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:53.500109 21213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:53.503958 21213 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40857
I20260620 19:11:53.504066 21280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40857 every 8 connection(s)
I20260620 19:11:53.504359 21213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:11:53.505379 21281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:53.507748 21281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505: Bootstrap starting.
I20260620 19:11:53.508531 21281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:53.508868 21281 log.cc:824] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:53.509002 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 21213
I20260620 19:11:53.509137 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:11:53.509326 812 external_mini_cluster.cc:1428] Setting key db6af31ab011c09c032f06e04ea83275
I20260620 19:11:53.509730 21281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505: No bootstrap required, opened a new log
I20260620 19:11:53.512205 21281 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfafce4c84dc4c65b14106ac45456505" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40857 } }
I20260620 19:11:53.512429 21281 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:53.512482 21281 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfafce4c84dc4c65b14106ac45456505, State: Initialized, Role: FOLLOWER
I20260620 19:11:53.512624 21281 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfafce4c84dc4c65b14106ac45456505" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40857 } }
I20260620 19:11:53.512727 21281 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:11:53.512758 21281 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:11:53.512813 21281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:53.512832 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:53.509895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49099 (local address 127.0.0.1:40857)
0620 19:11:53.510075 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:53.510086 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:53.510123 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:11:53.511292 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:53.511302 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:53.511340 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:53.511506 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:53.512150 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:53.512154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:53.512154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:53.512159 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:53.512203 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:53.512219 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:53.512295 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:53.512355 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 19:11:53.512395 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260620 19:11:53.513384 21281 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfafce4c84dc4c65b14106ac45456505" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40857 } }
I20260620 19:11:53.513521 21281 leader_election.cc:302] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfafce4c84dc4c65b14106ac45456505; no voters:
I20260620 19:11:53.513716 21281 leader_election.cc:288] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:11:53.513777 21286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:53.513857 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:11:53.513929 21286 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505 [term 1 LEADER]: Becoming Leader. State: Replica: dfafce4c84dc4c65b14106ac45456505, State: Running, Role: LEADER
I20260620 19:11:53.514014 21281 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:11:53.514020 21286 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfafce4c84dc4c65b14106ac45456505" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40857 } }
I20260620 19:11:53.514369 21286 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfafce4c84dc4c65b14106ac45456505" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfafce4c84dc4c65b14106ac45456505" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40857 } } }
I20260620 19:11:53.514490 21286 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505 [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:53.514680 21286 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfafce4c84dc4c65b14106ac45456505. Latest consensus state: current_term: 1 leader_uuid: "dfafce4c84dc4c65b14106ac45456505" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfafce4c84dc4c65b14106ac45456505" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40857 } } }
I20260620 19:11:53.514791 21286 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505 [sys.catalog]: This master's current role is: LEADER
W20260620 19:11:53.515853 21306 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:11:53.515915 21306 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:11:53.515975 21293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:11:53.516170 21293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:11:53.516839 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:53.515665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60665 (local address 127.0.0.1:40857)
0620 19:11:53.515701 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:53.515703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:53.515890 (+ 187us) server_negotiation.cc:407] Connection header received
0620 19:11:53.515943 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:53.515945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:53.515958 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:53.516005 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:53.516623 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:53.516624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:53.516624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:53.516627 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:53.516640 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:53.516643 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:53.516688 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:53.516740 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 19:11:53.516751 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:11:53.517858 21293 catalog_manager.cc:1347] Generated new cluster ID: afc77b8c67b244b4badd3d3378caa663
I20260620 19:11:53.517910 21293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:11:53.546613 21293 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:11:53.546829 21293 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:11:53.558382 21293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfafce4c84dc4c65b14106ac45456505: Generated new TSK 0
I20260620 19:11:53.558625 21293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:11:53.569039 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 21213
2026-06-20T19:11:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:11:53 dist-test-slave-wd30 krb5kdc[21312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:11:53 dist-test-slave-wd30 krb5kdc[21312](info): set up 2 sockets
Jun 20 19:11:53 dist-test-slave-wd30 krb5kdc[21312](info): commencing operation
krb5kdc: starting...
W20260620 19:11:54.603293 812 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:11:54 dist-test-slave-wd30 krb5kdc[21312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:11:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:11:54.720111 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34417
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:54.802174 21325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:54.802397 21325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:54.802423 21325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:54.803715 21325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:11:54.803761 21325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:54.803776 21325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:54.803819 21325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:11:54.803851 21325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:11:54.805320 21325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:54.805593 21325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:54.805830 21325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:54.811075 21330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:54.811096 21331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:54.811177 21333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:54.811180 21325 server_base.cc:1034] running on GCE node
I20260620 19:11:54.811532 21325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:54.811833 21325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:54.812985 21325 hybrid_clock.cc:648] HybridClock initialized: now 1781982714812952 us; error 47 us; skew 500 ppm
Jun 20 19:11:54 dist-test-slave-wd30 krb5kdc[21312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:54.815635 21325 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:11:54.816063 21325 webserver.cc:460] Webserver started at http://127.0.0.1:39259/ using document root <none> and password file <none>
I20260620 19:11:54.816303 21325 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:54.816368 21325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:54.816504 21325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:54.817581 21325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "65e1d5d68f3040eaa25bc2a71a6a10ea"
format_stamp: "Formatted at 2026-06-20 19:11:54 on dist-test-slave-wd30"
server_key: "e7699c0cf8f002f1cd87881a8ad2e4a6"
server_key_iv: "480f62721a7ccaade30b103117593d00"
server_key_version: "encryptionkey@0"
I20260620 19:11:54.817961 21325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "65e1d5d68f3040eaa25bc2a71a6a10ea"
format_stamp: "Formatted at 2026-06-20 19:11:54 on dist-test-slave-wd30"
server_key: "e7699c0cf8f002f1cd87881a8ad2e4a6"
server_key_iv: "480f62721a7ccaade30b103117593d00"
server_key_version: "encryptionkey@0"
I20260620 19:11:54.819317 21325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:11:54.820165 21340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:54.820343 21325 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:11:54.820436 21325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "65e1d5d68f3040eaa25bc2a71a6a10ea"
format_stamp: "Formatted at 2026-06-20 19:11:54 on dist-test-slave-wd30"
server_key: "e7699c0cf8f002f1cd87881a8ad2e4a6"
server_key_iv: "480f62721a7ccaade30b103117593d00"
server_key_version: "encryptionkey@0"
I20260620 19:11:54.820528 21325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:54.837739 21325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:54.840250 21325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:54.840425 21325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:54.844400 21325 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34417
I20260620 19:11:54.844462 21392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34417 every 8 connection(s)
I20260620 19:11:54.844857 21325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:11:54.845810 21393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:54.848290 21393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea: Bootstrap starting.
I20260620 19:11:54.848899 21393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:54.849220 21393 log.cc:824] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:54.849932 21393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea: No bootstrap required, opened a new log
I20260620 19:11:54.851603 21393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e1d5d68f3040eaa25bc2a71a6a10ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34417 } }
I20260620 19:11:54.851783 21393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:54.851835 21393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e1d5d68f3040eaa25bc2a71a6a10ea, State: Initialized, Role: FOLLOWER
I20260620 19:11:54.852035 21393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e1d5d68f3040eaa25bc2a71a6a10ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34417 } }
I20260620 19:11:54.852131 21393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:11:54.852173 21393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:11:54.852226 21393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:54.852810 21393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e1d5d68f3040eaa25bc2a71a6a10ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34417 } }
I20260620 19:11:54.852931 21393 leader_election.cc:302] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65e1d5d68f3040eaa25bc2a71a6a10ea; no voters:
I20260620 19:11:54.853121 21393 leader_election.cc:288] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:11:54.853166 21396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:54.853396 21396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea [term 1 LEADER]: Becoming Leader. State: Replica: 65e1d5d68f3040eaa25bc2a71a6a10ea, State: Running, Role: LEADER
I20260620 19:11:54.853418 21393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:11:54.853585 21396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e1d5d68f3040eaa25bc2a71a6a10ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34417 } }
I20260620 19:11:54.853948 21397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65e1d5d68f3040eaa25bc2a71a6a10ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e1d5d68f3040eaa25bc2a71a6a10ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34417 } } }
I20260620 19:11:54.854051 21397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:54.853957 21398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65e1d5d68f3040eaa25bc2a71a6a10ea. Latest consensus state: current_term: 1 leader_uuid: "65e1d5d68f3040eaa25bc2a71a6a10ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e1d5d68f3040eaa25bc2a71a6a10ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34417 } } }
I20260620 19:11:54.854140 21398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:54.854324 21400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:11:54.854815 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 21325
I20260620 19:11:54.854926 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:11:54.855155 812 external_mini_cluster.cc:1428] Setting key cd43b626d2da28dbe7ada230a0f8ce8c
I20260620 19:11:54.855739 21400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:11:54.857529 21400 catalog_manager.cc:1347] Generated new cluster ID: 770e20f9e85c40128ee0ceafd67125a1
I20260620 19:11:54.857599 21400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:11:54 dist-test-slave-wd30 krb5kdc[21312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:11:54.862094 21400 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:11:54.862798 21400 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:11:54.865710 21415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:54.855817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40317 (local address 127.0.0.1:34417)
0620 19:11:54.856047 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:54.856058 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:54.856261 (+ 203us) server_negotiation.cc:407] Connection header received
0620 19:11:54.856565 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:54.856574 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:54.856634 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:54.856781 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:54.857572 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:54.858538 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:54.859381 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:54.859817 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:54.859946 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:54.862492 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:54.862497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:54.862497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:54.862511 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:54.864140 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:54.864518 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:54.864522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:54.864522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:54.864624 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:54.864837 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:54.864838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:54.864838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:54.865063 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:54.865232 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:54.865343 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 19:11:54.865501 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":68,"threads_started":1}
I20260620 19:11:54.866672 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:11:54.873624 21400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65e1d5d68f3040eaa25bc2a71a6a10ea: Generated new TSK 0
I20260620 19:11:54.873819 21400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:11:54.878408 21415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:54.871489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48705 (local address 127.0.0.1:34417)
0620 19:11:54.871544 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:54.871546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:54.871746 (+ 200us) server_negotiation.cc:407] Connection header received
0620 19:11:54.871849 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:54.871850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:54.871860 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:54.871931 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:54.872485 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:54.873043 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:54.874090 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:54.874314 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:54.874392 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:54.875718 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:54.875720 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:54.875720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:54.875726 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:54.877209 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:54.877606 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:54.877607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:54.877607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:54.877670 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:54.877883 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:54.877884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:54.877884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:54.878006 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:54.878114 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:54.878215 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 19:11:54.878304 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":33}
I20260620 19:11:54.880550 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 21325
2026-06-20T19:11:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:11:54 dist-test-slave-wd30 krb5kdc[21424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:11:54 dist-test-slave-wd30 krb5kdc[21424](info): set up 2 sockets
Jun 20 19:11:54 dist-test-slave-wd30 krb5kdc[21424](info): commencing operation
krb5kdc: starting...
W20260620 19:11:55.911286 812 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:11:55 dist-test-slave-wd30 krb5kdc[21424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:11:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:11:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:11:56.029580 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:56.113288 21437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:56.113586 21437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:56.113633 21437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:56.114885 21437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:11:56.114967 21437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:56.114995 21437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:56.115026 21437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:11:56.115064 21437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:11:56.116624 21437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:56.116940 21437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:56.117187 21437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:11:56.122424 21437 server_base.cc:1034] running on GCE node
W20260620 19:11:56.122391 21443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:56.122388 21442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:56.122377 21445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:56.122941 21437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:56.123329 21437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:56.124513 21437 hybrid_clock.cc:648] HybridClock initialized: now 1781982716124508 us; error 38 us; skew 500 ppm
Jun 20 19:11:56 dist-test-slave-wd30 krb5kdc[21424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:56.130072 21437 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:11:56.130590 21437 webserver.cc:460] Webserver started at http://127.0.0.1:38209/ using document root <none> and password file <none>
I20260620 19:11:56.130831 21437 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:56.130905 21437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:56.131062 21437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:56.132200 21437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "12cdf00657ce456281a6898f59ebabff"
format_stamp: "Formatted at 2026-06-20 19:11:56 on dist-test-slave-wd30"
server_key: "477e1e67ae9c5a4b195a028e23ba6ec8"
server_key_iv: "9cc972fc87c7d5fe77bc624e5f4415b4"
server_key_version: "encryptionkey@0"
I20260620 19:11:56.132580 21437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "12cdf00657ce456281a6898f59ebabff"
format_stamp: "Formatted at 2026-06-20 19:11:56 on dist-test-slave-wd30"
server_key: "477e1e67ae9c5a4b195a028e23ba6ec8"
server_key_iv: "9cc972fc87c7d5fe77bc624e5f4415b4"
server_key_version: "encryptionkey@0"
I20260620 19:11:56.134006 21437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:11:56.134748 21452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:56.134927 21437 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:11:56.135035 21437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "12cdf00657ce456281a6898f59ebabff"
format_stamp: "Formatted at 2026-06-20 19:11:56 on dist-test-slave-wd30"
server_key: "477e1e67ae9c5a4b195a028e23ba6ec8"
server_key_iv: "9cc972fc87c7d5fe77bc624e5f4415b4"
server_key_version: "encryptionkey@0"
I20260620 19:11:56.135110 21437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:56.146798 21437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:56.148726 21437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:56.148926 21437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:56.153980 21437 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36419
I20260620 19:11:56.154012 21504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36419 every 8 connection(s)
I20260620 19:11:56.154376 21437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:11:56.154597 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 21437
I20260620 19:11:56.154708 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:11:56.154909 812 external_mini_cluster.cc:1428] Setting key 6d54344d84b67061337028a4099044e2
I20260620 19:11:56.155450 21505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:56.158126 21505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff: Bootstrap starting.
I20260620 19:11:56.158740 21505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:56.159036 21505 log.cc:824] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:56.159776 21505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff: No bootstrap required, opened a new log
I20260620 19:11:56.160291 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:56.155496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33335 (local address 127.0.0.1:36419)
0620 19:11:56.155670 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:56.155681 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:56.155717 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:11:56.156160 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:56.156166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:56.156216 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:56.156380 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:56.156560 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:56.157483 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:56.158305 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:56.158737 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:56.158868 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:56.159646 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:56.159650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:56.159650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:56.159839 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:56.159879 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:56.159901 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:56.159975 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:56.160051 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 19:11:56.160120 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260620 19:11:56.161423 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:11:56.162067 21505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cdf00657ce456281a6898f59ebabff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36419 } }
I20260620 19:11:56.162225 21505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:56.162287 21505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12cdf00657ce456281a6898f59ebabff, State: Initialized, Role: FOLLOWER
I20260620 19:11:56.162438 21505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cdf00657ce456281a6898f59ebabff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36419 } }
I20260620 19:11:56.162521 21505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:11:56.162563 21505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:11:56.162618 21505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:56.163283 21505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cdf00657ce456281a6898f59ebabff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36419 } }
I20260620 19:11:56.163406 21505 leader_election.cc:302] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12cdf00657ce456281a6898f59ebabff; no voters:
I20260620 19:11:56.163610 21505 leader_election.cc:288] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:11:56.163637 21515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:56.163744 21515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff [term 1 LEADER]: Becoming Leader. State: Replica: 12cdf00657ce456281a6898f59ebabff, State: Running, Role: LEADER
I20260620 19:11:56.163872 21505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:11:56.163888 21515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cdf00657ce456281a6898f59ebabff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36419 } }
I20260620 19:11:56.164211 21516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12cdf00657ce456281a6898f59ebabff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cdf00657ce456281a6898f59ebabff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36419 } } }
I20260620 19:11:56.164378 21516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:56.164201 21515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12cdf00657ce456281a6898f59ebabff. Latest consensus state: current_term: 1 leader_uuid: "12cdf00657ce456281a6898f59ebabff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12cdf00657ce456281a6898f59ebabff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36419 } } }
I20260620 19:11:56.164664 21515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:56.164832 21523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:11:56.165383 21523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:11:56.166266 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:56.162742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43283 (local address 127.0.0.1:36419)
0620 19:11:56.162774 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:56.162776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:56.162919 (+ 143us) server_negotiation.cc:407] Connection header received
0620 19:11:56.163000 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:56.163001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:56.163011 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:56.163056 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:56.163216 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:56.163922 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:56.164922 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:56.165237 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:56.165303 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:56.166012 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:56.166014 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:56.166014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:56.166019 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:56.166034 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:56.166039 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:56.166085 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:56.166155 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 19:11:56.166177 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:11:56.166914 21523 catalog_manager.cc:1347] Generated new cluster ID: be0faf68f1e2426388cf2e8b306f3b25
I20260620 19:11:56.166967 21523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:11:56.195237 21523 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:11:56.195848 21523 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:11:56.200873 21523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12cdf00657ce456281a6898f59ebabff: Generated new TSK 0
I20260620 19:11:56.201097 21523 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:11:56.209410 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 21437
2026-06-20T19:11:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:11:56 dist-test-slave-wd30 krb5kdc[21536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:11:56 dist-test-slave-wd30 krb5kdc[21536](info): set up 2 sockets
Jun 20 19:11:56 dist-test-slave-wd30 krb5kdc[21536](info): commencing operation
krb5kdc: starting...
W20260620 19:11:57.259307 812 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:11:57 dist-test-slave-wd30 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:11:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:11:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:11:57.376243 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:57.457495 21549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:57.457762 21549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:57.457803 21549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:57.459012 21549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:11:57.459084 21549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:57.459107 21549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:57.459129 21549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:11:57.459200 21549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:11:57.460784 21549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:57.461073 21549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:57.461311 21549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:57.466305 21555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:57.466315 21549 server_base.cc:1034] running on GCE node
W20260620 19:11:57.466282 21557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:57.466295 21554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:57.466835 21549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:57.467180 21549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:57.468339 21549 hybrid_clock.cc:648] HybridClock initialized: now 1781982717468319 us; error 43 us; skew 500 ppm
Jun 20 19:11:57 dist-test-slave-wd30 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:57.470912 21549 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:11:57.471410 21549 webserver.cc:460] Webserver started at http://127.0.0.1:40295/ using document root <none> and password file <none>
I20260620 19:11:57.471654 21549 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:57.471719 21549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:57.471858 21549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:57.472924 21549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "4973acc84be449c18b74dcb10931cc44"
format_stamp: "Formatted at 2026-06-20 19:11:57 on dist-test-slave-wd30"
server_key: "8a7d53a4f0b12d4b7e7762d079c84d83"
server_key_iv: "08929ac0785bcf740ab1d680dfd7e71d"
server_key_version: "encryptionkey@0"
I20260620 19:11:57.473292 21549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "4973acc84be449c18b74dcb10931cc44"
format_stamp: "Formatted at 2026-06-20 19:11:57 on dist-test-slave-wd30"
server_key: "8a7d53a4f0b12d4b7e7762d079c84d83"
server_key_iv: "08929ac0785bcf740ab1d680dfd7e71d"
server_key_version: "encryptionkey@0"
I20260620 19:11:57.474637 21549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:11:57.475847 21564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:57.476125 21549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:11:57.476265 21549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "4973acc84be449c18b74dcb10931cc44"
format_stamp: "Formatted at 2026-06-20 19:11:57 on dist-test-slave-wd30"
server_key: "8a7d53a4f0b12d4b7e7762d079c84d83"
server_key_iv: "08929ac0785bcf740ab1d680dfd7e71d"
server_key_version: "encryptionkey@0"
I20260620 19:11:57.476361 21549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:57.479938 21549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:57.482244 21549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:57.482396 21549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:57.486299 21549 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37105
I20260620 19:11:57.486335 21616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37105 every 8 connection(s)
I20260620 19:11:57.486701 21549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:11:57.487840 21617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:57.490170 21617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44: Bootstrap starting.
I20260620 19:11:57.490773 21617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:57.491098 21617 log.cc:824] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:57.491195 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 21549
I20260620 19:11:57.491298 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:11:57.491478 812 external_mini_cluster.cc:1428] Setting key a057798eda9b0761545d48fa53e267a9
I20260620 19:11:57.491926 21617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44: No bootstrap required, opened a new log
I20260620 19:11:57.494277 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:57.492058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38261 (local address 127.0.0.1:37105)
0620 19:11:57.492242 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:57.492253 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:57.492293 (+ 40us) server_negotiation.cc:407] Connection header received
0620 19:11:57.492871 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:57.492881 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:57.492934 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:57.493091 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:57.493816 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:57.493820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:57.493821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:57.493825 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:57.493871 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:57.493887 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:57.493982 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:57.494014 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 19:11:57.494055 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260620 19:11:57.495357 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:11:57.495406 21617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4973acc84be449c18b74dcb10931cc44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37105 } }
I20260620 19:11:57.495590 21617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:57.495639 21617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4973acc84be449c18b74dcb10931cc44, State: Initialized, Role: FOLLOWER
I20260620 19:11:57.495764 21617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4973acc84be449c18b74dcb10931cc44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37105 } }
I20260620 19:11:57.495849 21617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:11:57.495887 21617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:11:57.495937 21617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:57.496610 21617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4973acc84be449c18b74dcb10931cc44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37105 } }
I20260620 19:11:57.496742 21617 leader_election.cc:302] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4973acc84be449c18b74dcb10931cc44; no voters:
I20260620 19:11:57.496922 21617 leader_election.cc:288] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:11:57.496940 21627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:57.497102 21627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44 [term 1 LEADER]: Becoming Leader. State: Replica: 4973acc84be449c18b74dcb10931cc44, State: Running, Role: LEADER
I20260620 19:11:57.497223 21617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:11:57.497283 21627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4973acc84be449c18b74dcb10931cc44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37105 } }
I20260620 19:11:57.497608 21627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4973acc84be449c18b74dcb10931cc44. Latest consensus state: current_term: 1 leader_uuid: "4973acc84be449c18b74dcb10931cc44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4973acc84be449c18b74dcb10931cc44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37105 } } }
I20260620 19:11:57.497777 21627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44 [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:57.497952 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:57.496876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44665 (local address 127.0.0.1:37105)
0620 19:11:57.496926 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:57.496928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:57.496937 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:11:57.496982 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:57.496983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:57.496988 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:57.497033 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:57.497687 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:57.497688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:57.497689 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:11:57.497693 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:57.497717 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:11:57.497721 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:57.497752 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:57.497875 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 19:11:57.497892 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 19:11:57.497623 21629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4973acc84be449c18b74dcb10931cc44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4973acc84be449c18b74dcb10931cc44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37105 } } }
I20260620 19:11:57.498013 21629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44 [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:57.498539 21636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:11:57.499060 21636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:11:57.500727 21636 catalog_manager.cc:1347] Generated new cluster ID: 87e4e8278639479cab0320f4e0575c6b
I20260620 19:11:57.500804 21636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:11:57.504513 21636 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:11:57.504699 21636 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:11:57.510936 21636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4973acc84be449c18b74dcb10931cc44: Generated new TSK 0
I20260620 19:11:57.511173 21636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:11:57.515300 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 21549
2026-06-20T19:11:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:11:57 dist-test-slave-wd30 krb5kdc[21648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:11:57 dist-test-slave-wd30 krb5kdc[21648](info): set up 2 sockets
Jun 20 19:11:57 dist-test-slave-wd30 krb5kdc[21648](info): commencing operation
krb5kdc: starting...
W20260620 19:11:58.571254 812 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:11:58 dist-test-slave-wd30 krb5kdc[21648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:11:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:11:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:11:58.687930 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:58.771360 21661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:58.771651 21661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:58.771693 21661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:58.772856 21661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:11:58.772941 21661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:58.772970 21661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:58.772996 21661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:11:58.773039 21661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:11:58.774550 21661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:11:58.774835 21661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:58.775065 21661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:58.780046 21666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:58.780038 21667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:58.780004 21669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:58.780318 21661 server_base.cc:1034] running on GCE node
I20260620 19:11:58.780635 21661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:58.780961 21661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:58.782158 21661 hybrid_clock.cc:648] HybridClock initialized: now 1781982718782148 us; error 32 us; skew 500 ppm
Jun 20 19:11:58 dist-test-slave-wd30 krb5kdc[21648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:58.784724 21661 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:11:58.785220 21661 webserver.cc:460] Webserver started at http://127.0.0.1:33811/ using document root <none> and password file <none>
I20260620 19:11:58.785456 21661 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:58.785530 21661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:58.785686 21661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:11:58.786800 21661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "ea695c550bac4deca2864b8f2cc431ef"
format_stamp: "Formatted at 2026-06-20 19:11:58 on dist-test-slave-wd30"
server_key: "f1c0f414a843fbf75f427acf565c6861"
server_key_iv: "c95c1b5d79b556413e13191080e71eb0"
server_key_version: "encryptionkey@0"
I20260620 19:11:58.787241 21661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "ea695c550bac4deca2864b8f2cc431ef"
format_stamp: "Formatted at 2026-06-20 19:11:58 on dist-test-slave-wd30"
server_key: "f1c0f414a843fbf75f427acf565c6861"
server_key_iv: "c95c1b5d79b556413e13191080e71eb0"
server_key_version: "encryptionkey@0"
I20260620 19:11:58.788631 21661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:11:58.789382 21676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:58.789541 21661 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:11:58.789644 21661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "ea695c550bac4deca2864b8f2cc431ef"
format_stamp: "Formatted at 2026-06-20 19:11:58 on dist-test-slave-wd30"
server_key: "f1c0f414a843fbf75f427acf565c6861"
server_key_iv: "c95c1b5d79b556413e13191080e71eb0"
server_key_version: "encryptionkey@0"
I20260620 19:11:58.789729 21661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:58.808192 21661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:58.810211 21661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:58.810400 21661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:58.814884 21661 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35683
I20260620 19:11:58.814918 21728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35683 every 8 connection(s)
I20260620 19:11:58.815310 21661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:11:58.816200 21729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:58.818447 21729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef: Bootstrap starting.
I20260620 19:11:58.818989 21729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:58.819316 21729 log.cc:824] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:58.819986 21729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef: No bootstrap required, opened a new log
I20260620 19:11:58.821516 21729 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea695c550bac4deca2864b8f2cc431ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35683 } }
I20260620 19:11:58.821658 21729 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:58.821681 21729 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea695c550bac4deca2864b8f2cc431ef, State: Initialized, Role: FOLLOWER
I20260620 19:11:58.821799 21729 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea695c550bac4deca2864b8f2cc431ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35683 } }
I20260620 19:11:58.821854 21729 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:11:58.821875 21729 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:11:58.821902 21729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:58.822391 21729 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea695c550bac4deca2864b8f2cc431ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35683 } }
I20260620 19:11:58.822476 21729 leader_election.cc:302] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea695c550bac4deca2864b8f2cc431ef; no voters:
I20260620 19:11:58.822665 21729 leader_election.cc:288] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:11:58.822742 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 21661
I20260620 19:11:58.822750 21732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:58.822865 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:11:58.822914 21732 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef [term 1 LEADER]: Becoming Leader. State: Replica: ea695c550bac4deca2864b8f2cc431ef, State: Running, Role: LEADER
I20260620 19:11:58.822940 21729 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:11:58.823047 812 external_mini_cluster.cc:1428] Setting key dbeade3e8269d1dd756850e57c76424b
I20260620 19:11:58.822993 21732 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea695c550bac4deca2864b8f2cc431ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35683 } }
I20260620 19:11:58.823340 21732 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea695c550bac4deca2864b8f2cc431ef. Latest consensus state: current_term: 1 leader_uuid: "ea695c550bac4deca2864b8f2cc431ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea695c550bac4deca2864b8f2cc431ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35683 } } }
I20260620 19:11:58.823483 21732 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:58.823750 21741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:11:58.823804 21734 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea695c550bac4deca2864b8f2cc431ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea695c550bac4deca2864b8f2cc431ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35683 } } }
I20260620 19:11:58.823881 21734 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:58.824402 21741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:11:58.826100 21741 catalog_manager.cc:1347] Generated new cluster ID: 46ed9365694c42c89a1599269ebe21ce
I20260620 19:11:58.826152 21741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:11:58 dist-test-slave-wd30 krb5kdc[21648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:11:58.832628 21741 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:11:58.833285 21741 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:11:58.834408 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:58.823788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.188:49861 (local address 127.0.0.1:35683)
0620 19:11:58.823950 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:58.823960 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:58.824179 (+ 219us) server_negotiation.cc:407] Connection header received
0620 19:11:58.824319 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:58.824326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:58.824411 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:58.824545 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:58.825442 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:58.826681 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:58.827594 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:58.828012 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:58.828137 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:58.830423 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:58.830427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:58.830427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:58.830439 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:58.832833 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:58.833203 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:58.833206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:58.833206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:58.833299 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:58.833565 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:58.833567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:58.833567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:58.833782 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:58.833968 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:58.834094 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 19:11:58.834195 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260620 19:11:58.835350 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:11:58.837759 21741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea695c550bac4deca2864b8f2cc431ef: Generated new TSK 0
I20260620 19:11:58.837920 21741 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:11:58.843668 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:58.836650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.188:46467 (local address 127.0.0.1:35683)
0620 19:11:58.836686 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:58.836688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:58.837568 (+ 880us) server_negotiation.cc:407] Connection header received
0620 19:11:58.837592 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:58.837593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:58.837603 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:58.837663 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:58.838263 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:58.838781 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:58.839574 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:58.839801 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:58.839880 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:58.841106 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:58.841107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:58.841108 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:58.841115 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:58.842501 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:58.842871 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:58.842872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:58.842873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:58.842931 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:58.843169 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:58.843170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:58.843170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:58.843282 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:58.843389 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:58.843506 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 19:11:58.843567 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:11:58.845526 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 21661
W20260620 19:11:58.849418 21646 connection.cc:535] client connection to 127.0.0.1:35683 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35683): Connection reset by peer (error 104)
2026-06-20T19:11:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:11:58 dist-test-slave-wd30 krb5kdc[21760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:11:58 dist-test-slave-wd30 krb5kdc[21760](info): set up 2 sockets
Jun 20 19:11:58 dist-test-slave-wd30 krb5kdc[21760](info): commencing operation
krb5kdc: starting...
W20260620 19:11:59.907397 812 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:11:59 dist-test-slave-wd30 krb5kdc[21760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:11:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:11:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:12:00.025421 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38769
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:00.107625 21773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:00.107893 21773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:00.107931 21773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:00.109124 21773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:00.109202 21773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:00.109227 21773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:00.109251 21773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:00.109277 21773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:00.110785 21773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:00.111075 21773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:00.111335 21773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:00.116354 21778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:00.116357 21779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:00.116402 21773 server_base.cc:1034] running on GCE node
W20260620 19:12:00.116354 21781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:00.116775 21773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:00.117079 21773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:00.118227 21773 hybrid_clock.cc:648] HybridClock initialized: now 1781982720118208 us; error 36 us; skew 500 ppm
Jun 20 19:12:00 dist-test-slave-wd30 krb5kdc[21760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:00.120746 21773 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:12:00.121218 21773 webserver.cc:460] Webserver started at http://127.0.0.1:43557/ using document root <none> and password file <none>
I20260620 19:12:00.121460 21773 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:00.121510 21773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:00.121634 21773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:00.122702 21773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "f2b094e0ef274a6dac887472eca32d4d"
format_stamp: "Formatted at 2026-06-20 19:12:00 on dist-test-slave-wd30"
server_key: "5c66dca950343df6f889557d4f9de29c"
server_key_iv: "d170b7317f5dadc6da2cc2f380630e5f"
server_key_version: "encryptionkey@0"
I20260620 19:12:00.123059 21773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "f2b094e0ef274a6dac887472eca32d4d"
format_stamp: "Formatted at 2026-06-20 19:12:00 on dist-test-slave-wd30"
server_key: "5c66dca950343df6f889557d4f9de29c"
server_key_iv: "d170b7317f5dadc6da2cc2f380630e5f"
server_key_version: "encryptionkey@0"
I20260620 19:12:00.124424 21773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:12:00.125140 21788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:00.125299 21773 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:00.125401 21773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "f2b094e0ef274a6dac887472eca32d4d"
format_stamp: "Formatted at 2026-06-20 19:12:00 on dist-test-slave-wd30"
server_key: "5c66dca950343df6f889557d4f9de29c"
server_key_iv: "d170b7317f5dadc6da2cc2f380630e5f"
server_key_version: "encryptionkey@0"
I20260620 19:12:00.125474 21773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:00.147863 21773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:00.149959 21773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:00.150137 21773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:00.154364 21773 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38769
I20260620 19:12:00.154496 21840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38769 every 8 connection(s)
I20260620 19:12:00.154809 21773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:00.155748 21841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:00.158102 21841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d: Bootstrap starting.
I20260620 19:12:00.158769 21841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:00.159062 21841 log.cc:824] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:00.159921 21841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d: No bootstrap required, opened a new log
I20260620 19:12:00.160637 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 21773
I20260620 19:12:00.160758 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:12:00.160948 812 external_mini_cluster.cc:1428] Setting key 764cf6837a1e17dcd2a37f5765b7c8b6
I20260620 19:12:00.162254 21841 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b094e0ef274a6dac887472eca32d4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38769 } }
I20260620 19:12:00.162480 21841 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:00.162524 21841 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2b094e0ef274a6dac887472eca32d4d, State: Initialized, Role: FOLLOWER
I20260620 19:12:00.162714 21841 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b094e0ef274a6dac887472eca32d4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38769 } }
I20260620 19:12:00.162794 21841 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:00.162817 21841 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:00.162864 21841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:00.163522 21841 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b094e0ef274a6dac887472eca32d4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38769 } }
I20260620 19:12:00.163625 21841 leader_election.cc:302] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2b094e0ef274a6dac887472eca32d4d; no voters:
I20260620 19:12:00.163789 21841 leader_election.cc:288] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:00.163826 21846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:00.163978 21846 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d [term 1 LEADER]: Becoming Leader. State: Replica: f2b094e0ef274a6dac887472eca32d4d, State: Running, Role: LEADER
I20260620 19:12:00.164072 21841 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:00.164111 21846 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b094e0ef274a6dac887472eca32d4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38769 } }
I20260620 19:12:00.164510 21846 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2b094e0ef274a6dac887472eca32d4d. Latest consensus state: current_term: 1 leader_uuid: "f2b094e0ef274a6dac887472eca32d4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b094e0ef274a6dac887472eca32d4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38769 } } }
I20260620 19:12:00.164654 21846 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:00.164906 21855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:00.165604 21847 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2b094e0ef274a6dac887472eca32d4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b094e0ef274a6dac887472eca32d4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38769 } } }
I20260620 19:12:00.165693 21855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:00.165751 21847 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:00.167609 21855 catalog_manager.cc:1347] Generated new cluster ID: 442d8e99395f4bd19c20b014f9dbf1ee
I20260620 19:12:00.167671 21855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:00.167725 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:00.161572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.188:55441 (local address 127.0.0.1:38769)
0620 19:12:00.161809 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:00.161833 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:00.161868 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:12:00.162236 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:00.162244 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:00.162300 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:00.162454 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:00.162684 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:00.164240 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:00.165385 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:00.166004 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:00.166119 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:00.166929 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:00.166934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:00.166934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:00.167197 (+ 263us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:00.167235 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:00.167253 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:00.167351 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:00.167440 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 19:12:00.167471 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":111,"threads_started":1}
I20260620 19:12:00.172256 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:12:00.173828 21855 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:00.174571 21855 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:00.177662 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:00.173821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.188:49391 (local address 127.0.0.1:38769)
0620 19:12:00.173859 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:00.173861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:00.174119 (+ 258us) server_negotiation.cc:407] Connection header received
0620 19:12:00.174232 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:00.174233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:00.174244 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:00.174275 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:00.174498 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:00.175243 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:00.176271 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:00.176599 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:00.176667 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:00.177406 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:00.177407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:00.177407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:00.177411 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:00.177427 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:00.177431 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:00.177480 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:00.177558 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 19:12:00.177571 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:12:00.179733 21855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2b094e0ef274a6dac887472eca32d4d: Generated new TSK 0
I20260620 19:12:00.179901 21855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:12:00.185931 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 21773
W20260620 19:12:00.190212 21758 connection.cc:535] client connection to 127.0.0.1:38769 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38769): Connection reset by peer (error 104)
2026-06-20T19:12:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:00 dist-test-slave-wd30 krb5kdc[21872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:00 dist-test-slave-wd30 krb5kdc[21872](info): set up 2 sockets
Jun 20 19:12:00 dist-test-slave-wd30 krb5kdc[21872](info): commencing operation
krb5kdc: starting...
W20260620 19:12:01.223685 812 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:12:01 dist-test-slave-wd30 krb5kdc[21872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:12:01.341789 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:01.423195 21885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:01.423417 21885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:01.423442 21885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:01.424544 21885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:01.424582 21885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:01.424597 21885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:01.424610 21885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:01.424623 21885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:01.426137 21885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:01.426337 21885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:01.426512 21885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:01.431663 21893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:01.431682 21885 server_base.cc:1034] running on GCE node
W20260620 19:12:01.431682 21891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:01.431664 21890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:01.432170 21885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:01.432582 21885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:01.433761 21885 hybrid_clock.cc:648] HybridClock initialized: now 1781982721433756 us; error 35 us; skew 500 ppm
Jun 20 19:12:01 dist-test-slave-wd30 krb5kdc[21872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:01.439662 21885 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:12:01.440132 21885 webserver.cc:460] Webserver started at http://127.0.0.1:34599/ using document root <none> and password file <none>
I20260620 19:12:01.440359 21885 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:01.440426 21885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:01.440573 21885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:01.441665 21885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "3c47a5daf8814d0babb0fd7245f28b42"
format_stamp: "Formatted at 2026-06-20 19:12:01 on dist-test-slave-wd30"
server_key: "dd75bc94f6a1870f650f77a9de48ee49"
server_key_iv: "3420ae862270261da9c306a66e7b709c"
server_key_version: "encryptionkey@0"
I20260620 19:12:01.442039 21885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "3c47a5daf8814d0babb0fd7245f28b42"
format_stamp: "Formatted at 2026-06-20 19:12:01 on dist-test-slave-wd30"
server_key: "dd75bc94f6a1870f650f77a9de48ee49"
server_key_iv: "3420ae862270261da9c306a66e7b709c"
server_key_version: "encryptionkey@0"
I20260620 19:12:01.443490 21885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:12:01.444355 21900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:01.444558 21885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:01.444669 21885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "3c47a5daf8814d0babb0fd7245f28b42"
format_stamp: "Formatted at 2026-06-20 19:12:01 on dist-test-slave-wd30"
server_key: "dd75bc94f6a1870f650f77a9de48ee49"
server_key_iv: "3420ae862270261da9c306a66e7b709c"
server_key_version: "encryptionkey@0"
I20260620 19:12:01.444747 21885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:01.448431 21885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:01.450680 21885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:01.450832 21885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:01.454699 21885 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37879
I20260620 19:12:01.454731 21952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37879 every 8 connection(s)
I20260620 19:12:01.455070 21885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:01.456076 21953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:01.456507 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 21885
I20260620 19:12:01.456647 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:12:01.456825 812 external_mini_cluster.cc:1428] Setting key f75f96bedc8bad254f255d83f462c463
I20260620 19:12:01.459038 21953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42: Bootstrap starting.
I20260620 19:12:01.459220 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:01.457358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.188:57215 (local address 127.0.0.1:37879)
0620 19:12:01.457521 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:01.457529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:01.457554 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:12:01.457948 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:01.457954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:01.457979 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:01.458094 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:01.458714 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:01.458718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:01.458718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:01.458722 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:01.458755 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:01.458766 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:01.458819 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:01.458913 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 19:12:01.458969 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260620 19:12:01.459764 21953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:01.460035 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:12:01.460052 21953 log.cc:824] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:01.460697 21953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42: No bootstrap required, opened a new log
I20260620 19:12:01.462446 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:01.461458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.188:51885 (local address 127.0.0.1:37879)
0620 19:12:01.461493 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:01.461495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:01.461581 (+ 86us) server_negotiation.cc:407] Connection header received
0620 19:12:01.461621 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:01.461621 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:01.461626 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:01.461693 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:01.462224 (+ 531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:01.462225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:01.462226 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:01.462231 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:01.462244 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:01.462249 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:01.462324 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:01.462361 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 19:12:01.462374 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:12:01.463101 21953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c47a5daf8814d0babb0fd7245f28b42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } }
I20260620 19:12:01.463352 21953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:01.463403 21953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c47a5daf8814d0babb0fd7245f28b42, State: Initialized, Role: FOLLOWER
I20260620 19:12:01.463511 21953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c47a5daf8814d0babb0fd7245f28b42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } }
I20260620 19:12:01.463583 21953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:01.463604 21953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:01.463632 21953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:01.464118 21953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c47a5daf8814d0babb0fd7245f28b42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } }
I20260620 19:12:01.464203 21953 leader_election.cc:302] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c47a5daf8814d0babb0fd7245f28b42; no voters:
I20260620 19:12:01.464335 21953 leader_election.cc:288] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:01.464395 21963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:01.464578 21963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42 [term 1 LEADER]: Becoming Leader. State: Replica: 3c47a5daf8814d0babb0fd7245f28b42, State: Running, Role: LEADER
I20260620 19:12:01.464592 21953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:01.464690 21963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c47a5daf8814d0babb0fd7245f28b42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } }
I20260620 19:12:01.465078 21965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c47a5daf8814d0babb0fd7245f28b42. Latest consensus state: current_term: 1 leader_uuid: "3c47a5daf8814d0babb0fd7245f28b42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c47a5daf8814d0babb0fd7245f28b42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } } }
I20260620 19:12:01.465180 21965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:01.465072 21964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c47a5daf8814d0babb0fd7245f28b42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c47a5daf8814d0babb0fd7245f28b42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } } }
I20260620 19:12:01.465390 21964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:01.465384 21971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:01.466106 21971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:01.467623 21971 catalog_manager.cc:1347] Generated new cluster ID: 27f9b76e38e84cc9ad0f861c17f43570
I20260620 19:12:01.467672 21971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:01.475469 21971 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:01.475632 21971 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:01.481771 21971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c47a5daf8814d0babb0fd7245f28b42: Generated new TSK 0
I20260620 19:12:01.481966 21971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:12:01.486409 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 21885
2026-06-20T19:12:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:01 dist-test-slave-wd30 krb5kdc[21984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:01 dist-test-slave-wd30 krb5kdc[21984](info): set up 2 sockets
Jun 20 19:12:01 dist-test-slave-wd30 krb5kdc[21984](info): commencing operation
krb5kdc: starting...
W20260620 19:12:02.527256 812 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:12:02 dist-test-slave-wd30 krb5kdc[21984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:12:02.645901 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41443
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:02.727896 21997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:02.728113 21997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:02.728137 21997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:02.729352 21997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:02.729398 21997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:02.729413 21997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:02.729425 21997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:02.729437 21997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:02.731014 21997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41443
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:02.731295 21997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:02.731525 21997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:02.736527 22005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:02.736575 22002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:02.736560 22003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:02.736580 21997 server_base.cc:1034] running on GCE node
I20260620 19:12:02.736940 21997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:02.737234 21997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:02.738396 21997 hybrid_clock.cc:648] HybridClock initialized: now 1781982722738389 us; error 33 us; skew 500 ppm
Jun 20 19:12:02 dist-test-slave-wd30 krb5kdc[21984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:02.740906 21997 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:12:02.741334 21997 webserver.cc:460] Webserver started at http://127.0.0.1:40947/ using document root <none> and password file <none>
I20260620 19:12:02.741551 21997 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:02.741596 21997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:02.741765 21997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:02.742851 21997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "5aa3900b978847bebde1970f96b52c3d"
format_stamp: "Formatted at 2026-06-20 19:12:02 on dist-test-slave-wd30"
server_key: "8df8bf72ea471e24390c8172add8dd7a"
server_key_iv: "2d79b8678048ef9b9feaebcefb9e1401"
server_key_version: "encryptionkey@0"
I20260620 19:12:02.743264 21997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "5aa3900b978847bebde1970f96b52c3d"
format_stamp: "Formatted at 2026-06-20 19:12:02 on dist-test-slave-wd30"
server_key: "8df8bf72ea471e24390c8172add8dd7a"
server_key_iv: "2d79b8678048ef9b9feaebcefb9e1401"
server_key_version: "encryptionkey@0"
I20260620 19:12:02.744588 21997 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:02.745283 22012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:02.745448 21997 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:02.745544 21997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "5aa3900b978847bebde1970f96b52c3d"
format_stamp: "Formatted at 2026-06-20 19:12:02 on dist-test-slave-wd30"
server_key: "8df8bf72ea471e24390c8172add8dd7a"
server_key_iv: "2d79b8678048ef9b9feaebcefb9e1401"
server_key_version: "encryptionkey@0"
I20260620 19:12:02.745627 21997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:02.763463 21997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:02.765441 21997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:02.765714 21997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:02.769744 21997 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41443
I20260620 19:12:02.769785 22064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41443 every 8 connection(s)
I20260620 19:12:02.770162 21997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:02.770802 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 21997
I20260620 19:12:02.770927 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:12:02.771103 812 external_mini_cluster.cc:1428] Setting key a7d29558c06d340e1326ab5887f2f750
I20260620 19:12:02.771152 22065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:02.774391 22065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d: Bootstrap starting.
I20260620 19:12:02.775072 22065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:02.775420 22065 log.cc:824] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:02.776144 22065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d: No bootstrap required, opened a new log
Jun 20 19:12:02 dist-test-slave-wd30 krb5kdc[21984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:12:02.777976 22065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa3900b978847bebde1970f96b52c3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41443 } }
I20260620 19:12:02.778180 22065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:02.778234 22065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa3900b978847bebde1970f96b52c3d, State: Initialized, Role: FOLLOWER
I20260620 19:12:02.778391 22065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa3900b978847bebde1970f96b52c3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41443 } }
I20260620 19:12:02.778482 22065 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:02.778541 22065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:02.778596 22065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:02.779111 22065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa3900b978847bebde1970f96b52c3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41443 } }
I20260620 19:12:02.779281 22065 leader_election.cc:302] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aa3900b978847bebde1970f96b52c3d; no voters:
I20260620 19:12:02.779455 22065 leader_election.cc:288] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:02.779507 22070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:02.779639 22070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d [term 1 LEADER]: Becoming Leader. State: Replica: 5aa3900b978847bebde1970f96b52c3d, State: Running, Role: LEADER
I20260620 19:12:02.779716 22065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:02.779855 22070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa3900b978847bebde1970f96b52c3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41443 } }
I20260620 19:12:02.780174 22070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aa3900b978847bebde1970f96b52c3d. Latest consensus state: current_term: 1 leader_uuid: "5aa3900b978847bebde1970f96b52c3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa3900b978847bebde1970f96b52c3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41443 } } }
I20260620 19:12:02.780251 22070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:02.780196 22071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5aa3900b978847bebde1970f96b52c3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa3900b978847bebde1970f96b52c3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41443 } } }
I20260620 19:12:02.780426 22071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:02.780923 22076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:02.781497 22076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:02.782286 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:02.771645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.188:53729 (local address 127.0.0.1:41443)
0620 19:12:02.771824 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:02.771832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:02.771857 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:12:02.772261 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:02.772268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:02.772317 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:02.772428 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:02.773272 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:02.774135 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:02.775240 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:02.775676 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:02.775770 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:02.778069 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:02.778074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:02.778074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:02.778086 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:02.780163 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:02.780679 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:02.780681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:02.780681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:02.780760 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:02.781006 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:02.781007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:02.781008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:02.781311 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:02.781457 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:02.781987 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 19:12:02.782084 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260620 19:12:02.783326 22076 catalog_manager.cc:1347] Generated new cluster ID: bc25286ce57b4b55907147fd19191e21
I20260620 19:12:02.783393 22076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:02.783644 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:12:02.792608 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:02.785215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.188:33897 (local address 127.0.0.1:41443)
0620 19:12:02.785246 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:02.785248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:02.785255 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:12:02.785321 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:02.785322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:02.785330 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:02.785380 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:02.786017 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:02.786739 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:02.787838 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:02.788156 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:02.788222 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:02.789713 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:02.789715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:02.789715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:02.789722 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:02.791350 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:02.791785 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:02.791786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:02.791787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:02.791843 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:02.792078 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:02.792078 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:02.792079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:02.792198 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:02.792289 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:02.792452 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 19:12:02.792513 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:12:02.803089 22076 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:02.803890 22076 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:02.809401 22076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aa3900b978847bebde1970f96b52c3d: Generated new TSK 0
I20260620 19:12:02.809607 22076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:12:02.812701 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 21997
W20260620 19:12:02.817250 21982 connection.cc:535] client connection to 127.0.0.1:41443 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41443): Connection reset by peer (error 104)
2026-06-20T19:12:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:02 dist-test-slave-wd30 krb5kdc[22096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:02 dist-test-slave-wd30 krb5kdc[22096](info): set up 2 sockets
Jun 20 19:12:02 dist-test-slave-wd30 krb5kdc[22096](info): commencing operation
krb5kdc: starting...
W20260620 19:12:03.871357 812 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:12:03 dist-test-slave-wd30 krb5kdc[22096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:12:03.989802 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39331
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:04.070753 22109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:04.071000 22109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:04.071039 22109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:04.072248 22109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:04.072317 22109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:04.072345 22109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:04.072376 22109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:04.072398 22109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:04.073906 22109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39331
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:04.074184 22109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:04.074420 22109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:04.079267 22114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:04.079284 22115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:04.079362 22109 server_base.cc:1034] running on GCE node
W20260620 19:12:04.079289 22117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:04.079754 22109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:04.080062 22109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:04.081240 22109 hybrid_clock.cc:648] HybridClock initialized: now 1781982724081218 us; error 33 us; skew 500 ppm
Jun 20 19:12:04 dist-test-slave-wd30 krb5kdc[22096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:04.083664 22109 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:12:04.084097 22109 webserver.cc:460] Webserver started at http://127.0.0.1:41343/ using document root <none> and password file <none>
I20260620 19:12:04.084343 22109 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:04.084410 22109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:04.084554 22109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:04.085649 22109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "3635e5aeff624cfdafaff7dbc73e12e3"
format_stamp: "Formatted at 2026-06-20 19:12:04 on dist-test-slave-wd30"
server_key: "81afe723bd3ec4fe8ddf82964fe9311a"
server_key_iv: "36b704984ae09a055abad2058bcdcdc3"
server_key_version: "encryptionkey@0"
I20260620 19:12:04.086011 22109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "3635e5aeff624cfdafaff7dbc73e12e3"
format_stamp: "Formatted at 2026-06-20 19:12:04 on dist-test-slave-wd30"
server_key: "81afe723bd3ec4fe8ddf82964fe9311a"
server_key_iv: "36b704984ae09a055abad2058bcdcdc3"
server_key_version: "encryptionkey@0"
I20260620 19:12:04.087405 22109 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:12:04.088090 22124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:04.088253 22109 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:04.088362 22109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "3635e5aeff624cfdafaff7dbc73e12e3"
format_stamp: "Formatted at 2026-06-20 19:12:04 on dist-test-slave-wd30"
server_key: "81afe723bd3ec4fe8ddf82964fe9311a"
server_key_iv: "36b704984ae09a055abad2058bcdcdc3"
server_key_version: "encryptionkey@0"
I20260620 19:12:04.088434 22109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:04.110602 22109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:04.112538 22109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:04.112711 22109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:04.116727 22109 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39331
I20260620 19:12:04.116775 22176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39331 every 8 connection(s)
I20260620 19:12:04.117136 22109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:04.118104 22177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:04.120419 22177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3: Bootstrap starting.
I20260620 19:12:04.120992 22177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:04.121282 22177 log.cc:824] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:04.121978 22177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3: No bootstrap required, opened a new log
I20260620 19:12:04.123581 22177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3635e5aeff624cfdafaff7dbc73e12e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39331 } }
I20260620 19:12:04.123723 22177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:04.123746 22177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3635e5aeff624cfdafaff7dbc73e12e3, State: Initialized, Role: FOLLOWER
I20260620 19:12:04.123931 22177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3635e5aeff624cfdafaff7dbc73e12e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39331 } }
I20260620 19:12:04.124014 22177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:04.124061 22177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:04.124112 22177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:04.124640 22177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3635e5aeff624cfdafaff7dbc73e12e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39331 } }
I20260620 19:12:04.124732 22177 leader_election.cc:302] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3635e5aeff624cfdafaff7dbc73e12e3; no voters:
I20260620 19:12:04.124943 22177 leader_election.cc:288] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:04.124979 22180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:04.124974 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 22109
I20260620 19:12:04.125090 22180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3 [term 1 LEADER]: Becoming Leader. State: Replica: 3635e5aeff624cfdafaff7dbc73e12e3, State: Running, Role: LEADER
I20260620 19:12:04.125099 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:12:04.125212 22177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:04.125278 812 external_mini_cluster.cc:1428] Setting key ab85cd099714eed4a7f5a8bc65c31b30
I20260620 19:12:04.125247 22180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3635e5aeff624cfdafaff7dbc73e12e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39331 } }
I20260620 19:12:04.125555 22180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3635e5aeff624cfdafaff7dbc73e12e3. Latest consensus state: current_term: 1 leader_uuid: "3635e5aeff624cfdafaff7dbc73e12e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3635e5aeff624cfdafaff7dbc73e12e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39331 } } }
I20260620 19:12:04.125680 22180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:04.125730 22181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3635e5aeff624cfdafaff7dbc73e12e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3635e5aeff624cfdafaff7dbc73e12e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39331 } } }
I20260620 19:12:04.125880 22181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:04.126065 22188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:04.126683 22188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:04.129505 22188 catalog_manager.cc:1347] Generated new cluster ID: 8e9dce27926641dd8d4310d5f2090ecb
I20260620 19:12:04.129561 22188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:04.131973 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:04.125939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.188:37831 (local address 127.0.0.1:39331)
0620 19:12:04.126118 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:04.126127 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:04.126865 (+ 738us) server_negotiation.cc:407] Connection header received
0620 19:12:04.127098 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:04.127105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:04.127212 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:04.127339 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:04.127674 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:04.129047 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:04.130014 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:04.130469 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:04.130589 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:04.131351 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:04.131355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:04.131355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:04.131532 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:04.131561 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:04.131572 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:04.131665 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:04.131752 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 19:12:04.131783 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260620 19:12:04.132844 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:12:04.136013 22188 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:04.136706 22188 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:04.138499 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:04.135421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.188:54691 (local address 127.0.0.1:39331)
0620 19:12:04.135464 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:04.135466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:04.135678 (+ 212us) server_negotiation.cc:407] Connection header received
0620 19:12:04.135727 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:04.135728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:04.135751 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:04.135814 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:04.135970 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:04.136511 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:04.137303 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:04.137519 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:04.137598 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:04.138273 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:04.138274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:04.138275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:04.138280 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:04.138295 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:04.138299 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:04.138352 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:04.138412 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 19:12:04.138423 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:12:04.151737 22188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3635e5aeff624cfdafaff7dbc73e12e3: Generated new TSK 0
I20260620 19:12:04.151975 22188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:12:04.156595 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 22109
W20260620 19:12:04.160939 22094 connection.cc:535] client connection to 127.0.0.1:39331 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39331): Connection reset by peer (error 104)
2026-06-20T19:12:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:04 dist-test-slave-wd30 krb5kdc[22208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:04 dist-test-slave-wd30 krb5kdc[22208](info): set up 2 sockets
Jun 20 19:12:04 dist-test-slave-wd30 krb5kdc[22208](info): commencing operation
krb5kdc: starting...
W20260620 19:12:05.215628 812 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:12:05 dist-test-slave-wd30 krb5kdc[22208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:12:05.334156 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:05.415006 22221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:05.415249 22221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:05.415274 22221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:05.416464 22221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:05.416510 22221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:05.416524 22221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:05.416538 22221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:05.416549 22221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:05.418066 22221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35977
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:05.418329 22221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:05.418561 22221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:05.423430 22226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:05.423568 22229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:05.423411 22227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:05.423643 22221 server_base.cc:1034] running on GCE node
I20260620 19:12:05.423995 22221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:05.424278 22221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:05.425463 22221 hybrid_clock.cc:648] HybridClock initialized: now 1781982725425452 us; error 34 us; skew 500 ppm
Jun 20 19:12:05 dist-test-slave-wd30 krb5kdc[22208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:05.428081 22221 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:12:05.428571 22221 webserver.cc:460] Webserver started at http://127.0.0.1:40307/ using document root <none> and password file <none>
I20260620 19:12:05.428851 22221 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:05.428925 22221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:05.429093 22221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:05.430234 22221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "857129db9c154aea83a3a5f44cce8696"
format_stamp: "Formatted at 2026-06-20 19:12:05 on dist-test-slave-wd30"
server_key: "2415ec4b2ff428223e4b9018ab23b242"
server_key_iv: "65aefcf2e4e7d2b88af7263a1ebe4a1d"
server_key_version: "encryptionkey@0"
I20260620 19:12:05.430635 22221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "857129db9c154aea83a3a5f44cce8696"
format_stamp: "Formatted at 2026-06-20 19:12:05 on dist-test-slave-wd30"
server_key: "2415ec4b2ff428223e4b9018ab23b242"
server_key_iv: "65aefcf2e4e7d2b88af7263a1ebe4a1d"
server_key_version: "encryptionkey@0"
I20260620 19:12:05.432205 22221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:12:05.433094 22236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:05.433301 22221 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:05.433408 22221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "857129db9c154aea83a3a5f44cce8696"
format_stamp: "Formatted at 2026-06-20 19:12:05 on dist-test-slave-wd30"
server_key: "2415ec4b2ff428223e4b9018ab23b242"
server_key_iv: "65aefcf2e4e7d2b88af7263a1ebe4a1d"
server_key_version: "encryptionkey@0"
I20260620 19:12:05.433511 22221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:05.437252 22221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:05.439365 22221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:05.439545 22221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:05.443682 22221 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35977
I20260620 19:12:05.443710 22288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35977 every 8 connection(s)
I20260620 19:12:05.444128 22221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:05.445099 22289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:05.447399 22289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696: Bootstrap starting.
I20260620 19:12:05.447922 22289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:05.448159 22289 log.cc:824] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:05.448805 22289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696: No bootstrap required, opened a new log
I20260620 19:12:05.449100 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 22221
I20260620 19:12:05.449234 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:12:05.449425 812 external_mini_cluster.cc:1428] Setting key 0e3fc66105de02081461ba3281099868
I20260620 19:12:05.451017 22289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857129db9c154aea83a3a5f44cce8696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35977 } }
I20260620 19:12:05.451267 22289 raft_consensus.cc:374] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:05.451305 22289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857129db9c154aea83a3a5f44cce8696, State: Initialized, Role: FOLLOWER
I20260620 19:12:05.451480 22289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857129db9c154aea83a3a5f44cce8696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35977 } }
I20260620 19:12:05.451574 22289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:05.451612 22289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:05.451660 22289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:05.452382 22289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857129db9c154aea83a3a5f44cce8696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35977 } }
I20260620 19:12:05.452491 22289 leader_election.cc:302] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 857129db9c154aea83a3a5f44cce8696; no voters:
I20260620 19:12:05.452638 22289 leader_election.cc:288] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:05.452700 22294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:05.452956 22289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:05.452963 22294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696 [term 1 LEADER]: Becoming Leader. State: Replica: 857129db9c154aea83a3a5f44cce8696, State: Running, Role: LEADER
I20260620 19:12:05.453023 22293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:05.449993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.188:38407 (local address 127.0.0.1:35977)
0620 19:12:05.450204 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:05.450213 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:05.450248 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:12:05.451253 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:05.451263 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:05.451298 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:05.451443 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:05.452500 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:05.452505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:05.452505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:05.452511 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:05.452562 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:05.452579 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:05.452678 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:05.452756 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 19:12:05.452804 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":92,"threads_started":1}
I20260620 19:12:05.453142 22294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857129db9c154aea83a3a5f44cce8696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35977 } }
I20260620 19:12:05.453444 22294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 857129db9c154aea83a3a5f44cce8696. Latest consensus state: current_term: 1 leader_uuid: "857129db9c154aea83a3a5f44cce8696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857129db9c154aea83a3a5f44cce8696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35977 } } }
I20260620 19:12:05.453593 22294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:05.453624 22296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "857129db9c154aea83a3a5f44cce8696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857129db9c154aea83a3a5f44cce8696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35977 } } }
I20260620 19:12:05.453783 22296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:05.453915 22303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:05.454128 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:12:05.454474 22303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:05.456233 22303 catalog_manager.cc:1347] Generated new cluster ID: 0f3ad768c6214edc8832f25de847d43c
I20260620 19:12:05.456306 22303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:05.457129 22293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:05.455767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.188:50557 (local address 127.0.0.1:35977)
0620 19:12:05.455807 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:05.455809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:05.456046 (+ 237us) server_negotiation.cc:407] Connection header received
0620 19:12:05.456158 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:05.456159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:05.456165 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:05.456197 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:05.456886 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:05.456887 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:05.456888 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:05.456892 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:05.456906 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:05.456910 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:05.456968 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:05.457051 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 19:12:05.457064 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:12:05.470858 22303 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:05.471063 22303 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:05.474687 22303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 857129db9c154aea83a3a5f44cce8696: Generated new TSK 0
I20260620 19:12:05.474874 22303 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:12:05.485554 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 22221
2026-06-20T19:12:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:05 dist-test-slave-wd30 krb5kdc[22320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:05 dist-test-slave-wd30 krb5kdc[22320](info): set up 2 sockets
Jun 20 19:12:05 dist-test-slave-wd30 krb5kdc[22320](info): commencing operation
krb5kdc: starting...
W20260620 19:12:06.535353 812 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:12:06 dist-test-slave-wd30 krb5kdc[22320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:12:06.654693 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:39923
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:39923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:06.736784 22333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:06.737046 22333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:06.737084 22333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:06.738260 22333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:06.738328 22333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:06.738354 22333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:06.738384 22333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:06.738409 22333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:06.739952 22333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:39923
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:39923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:06.740237 22333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:06.740463 22333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:06.745460 22338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:06.745460 22333 server_base.cc:1034] running on GCE node
W20260620 19:12:06.745535 22341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:06.745515 22339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:06.745931 22333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:06.746254 22333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:06.747416 22333 hybrid_clock.cc:648] HybridClock initialized: now 1781982726747404 us; error 43 us; skew 500 ppm
Jun 20 19:12:06 dist-test-slave-wd30 krb5kdc[22320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:06.752782 22333 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:12:06.753293 22333 webserver.cc:460] Webserver started at http://127.0.203.62:36313/ using document root <none> and password file <none>
I20260620 19:12:06.753521 22333 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:06.753587 22333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:06.753739 22333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:06.754868 22333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "d883e57421284d0a87e0f6032a20dda4"
format_stamp: "Formatted at 2026-06-20 19:12:06 on dist-test-slave-wd30"
server_key: "3d7030279c4ef54b6b94692eba2a35fa"
server_key_iv: "930b5ad8e448e47fb2a436d5baccb46b"
server_key_version: "encryptionkey@0"
I20260620 19:12:06.755259 22333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "d883e57421284d0a87e0f6032a20dda4"
format_stamp: "Formatted at 2026-06-20 19:12:06 on dist-test-slave-wd30"
server_key: "3d7030279c4ef54b6b94692eba2a35fa"
server_key_iv: "930b5ad8e448e47fb2a436d5baccb46b"
server_key_version: "encryptionkey@0"
I20260620 19:12:06.756614 22333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:12:06.757370 22348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:06.757570 22333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:12:06.757670 22333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "d883e57421284d0a87e0f6032a20dda4"
format_stamp: "Formatted at 2026-06-20 19:12:06 on dist-test-slave-wd30"
server_key: "3d7030279c4ef54b6b94692eba2a35fa"
server_key_iv: "930b5ad8e448e47fb2a436d5baccb46b"
server_key_version: "encryptionkey@0"
I20260620 19:12:06.757761 22333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:06.774109 22333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:06.776489 22333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:06.776675 22333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:06.780822 22333 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:39923
I20260620 19:12:06.780859 22400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:39923 every 8 connection(s)
I20260620 19:12:06.781214 22333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:06.782068 22401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:06.784318 22401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4: Bootstrap starting.
I20260620 19:12:06.784832 22401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:06.785072 22401 log.cc:824] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:06.785720 22401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4: No bootstrap required, opened a new log
I20260620 19:12:06.787376 22401 raft_consensus.cc:348] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d883e57421284d0a87e0f6032a20dda4" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39923 } }
I20260620 19:12:06.787518 22401 raft_consensus.cc:374] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:06.787542 22401 raft_consensus.cc:729] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d883e57421284d0a87e0f6032a20dda4, State: Initialized, Role: FOLLOWER
I20260620 19:12:06.787644 22401 consensus_queue.cc:260] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d883e57421284d0a87e0f6032a20dda4" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39923 } }
I20260620 19:12:06.787698 22401 raft_consensus.cc:388] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:06.787719 22401 raft_consensus.cc:482] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:06.787745 22401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:06.788232 22401 raft_consensus.cc:504] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d883e57421284d0a87e0f6032a20dda4" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39923 } }
I20260620 19:12:06.788314 22401 leader_election.cc:302] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d883e57421284d0a87e0f6032a20dda4; no voters:
I20260620 19:12:06.788434 22401 leader_election.cc:288] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:06.788501 22404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:06.788663 22404 raft_consensus.cc:686] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4 [term 1 LEADER]: Becoming Leader. State: Replica: d883e57421284d0a87e0f6032a20dda4, State: Running, Role: LEADER
I20260620 19:12:06.788720 22401 sys_catalog.cc:564] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:06.788798 22404 consensus_queue.cc:237] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d883e57421284d0a87e0f6032a20dda4" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39923 } }
I20260620 19:12:06.789222 22406 sys_catalog.cc:455] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d883e57421284d0a87e0f6032a20dda4. Latest consensus state: current_term: 1 leader_uuid: "d883e57421284d0a87e0f6032a20dda4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d883e57421284d0a87e0f6032a20dda4" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39923 } } }
I20260620 19:12:06.789523 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 22333
I20260620 19:12:06.789675 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:12:06.789367 22405 sys_catalog.cc:455] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d883e57421284d0a87e0f6032a20dda4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d883e57421284d0a87e0f6032a20dda4" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39923 } } }
I20260620 19:12:06.789896 812 external_mini_cluster.cc:1428] Setting key 175a1a0db664df6141be430490001fd0
I20260620 19:12:06.789909 22405 sys_catalog.cc:458] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:06.790058 22406 sys_catalog.cc:458] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:06.790216 22419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:06.790907 22419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:06.792603 22419 catalog_manager.cc:1347] Generated new cluster ID: 8d536b69770e4c07953bd225ec9afc7f
I20260620 19:12:06.792657 22419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:12:06 dist-test-slave-wd30 krb5kdc[22320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:12:06.801890 22422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:06.790460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38147 (local address 127.0.203.62:39923)
0620 19:12:06.790647 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:06.790658 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:06.790695 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:12:06.790816 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:06.790823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:06.790879 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:06.791045 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:06.791936 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:06.793286 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:06.794370 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:06.794955 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:06.795053 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:06.797963 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:06.797968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:06.797969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:06.797984 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:06.800013 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:06.800445 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:06.800449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:06.800449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:06.800585 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:06.800817 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:06.800818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:06.800818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:06.801147 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:06.801304 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:06.801420 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 19:12:06.801655 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260620 19:12:06.803104 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:12:06.811797 22419 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:06.812538 22419 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:06.814500 22422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:06.804593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45799 (local address 127.0.203.62:39923)
0620 19:12:06.804627 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:06.804629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:06.807293 (+ 2664us) server_negotiation.cc:407] Connection header received
0620 19:12:06.807326 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:06.807327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:06.807339 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:06.807404 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:06.808005 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:06.808816 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:06.809795 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:06.810122 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:06.810195 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:06.811690 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:06.811692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:06.811692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:06.811699 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:06.813278 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:06.813718 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:06.813718 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:06.813719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:06.813772 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:06.814001 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:06.814001 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:06.814002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:06.814117 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:06.814234 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:06.814349 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 19:12:06.814404 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:12:06.816416 22419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d883e57421284d0a87e0f6032a20dda4: Generated new TSK 0
I20260620 19:12:06.816615 22419 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:12:06.819053 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 22333
W20260620 19:12:06.823632 22318 connection.cc:535] client connection to 127.0.203.62:39923 recv error: Network error: failed to read from TLS socket (remote: 127.0.203.62:39923): Connection reset by peer (error 104)
2026-06-20T19:12:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:06 dist-test-slave-wd30 krb5kdc[22432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:06 dist-test-slave-wd30 krb5kdc[22432](info): set up 2 sockets
Jun 20 19:12:06 dist-test-slave-wd30 krb5kdc[22432](info): commencing operation
krb5kdc: starting...
W20260620 19:12:07.855683 812 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:12:07 dist-test-slave-wd30 krb5kdc[22432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:12:07.973541 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:35849
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:45985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:35849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:08.055326 22445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:08.055541 22445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:08.055567 22445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:08.056747 22445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:08.056797 22445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:08.056813 22445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:08.056828 22445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:08.056841 22445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:08.058346 22445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:45985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:35849
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:35849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:08.058599 22445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:08.058835 22445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:08.064025 22450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:08.064060 22453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:08.064085 22451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:08.064275 22445 server_base.cc:1034] running on GCE node
I20260620 19:12:08.064508 22445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:08.064752 22445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:08.065902 22445 hybrid_clock.cc:648] HybridClock initialized: now 1781982728065889 us; error 27 us; skew 500 ppm
Jun 20 19:12:08 dist-test-slave-wd30 krb5kdc[22432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:08.068495 22445 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:12:08.068976 22445 webserver.cc:460] Webserver started at http://127.0.203.62:37807/ using document root <none> and password file <none>
I20260620 19:12:08.069242 22445 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:08.069298 22445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:08.069485 22445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:08.070580 22445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "2d525d697c374f7bb8250499af145ad3"
format_stamp: "Formatted at 2026-06-20 19:12:08 on dist-test-slave-wd30"
server_key: "630c6ded48b388f39074d46978d04bed"
server_key_iv: "bc75d82d710f160afe47c0b46c0106ed"
server_key_version: "encryptionkey@0"
I20260620 19:12:08.070955 22445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "2d525d697c374f7bb8250499af145ad3"
format_stamp: "Formatted at 2026-06-20 19:12:08 on dist-test-slave-wd30"
server_key: "630c6ded48b388f39074d46978d04bed"
server_key_iv: "bc75d82d710f160afe47c0b46c0106ed"
server_key_version: "encryptionkey@0"
I20260620 19:12:08.072480 22445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:12:08.073340 22460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:08.073568 22445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:08.073679 22445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "2d525d697c374f7bb8250499af145ad3"
format_stamp: "Formatted at 2026-06-20 19:12:08 on dist-test-slave-wd30"
server_key: "630c6ded48b388f39074d46978d04bed"
server_key_iv: "bc75d82d710f160afe47c0b46c0106ed"
server_key_version: "encryptionkey@0"
I20260620 19:12:08.073770 22445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:08.083323 22445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:08.085335 22445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:08.085513 22445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:08.089713 22445 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:35849
I20260620 19:12:08.089773 22512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:35849 every 8 connection(s)
I20260620 19:12:08.090157 22445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:08.091234 22513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:08.093500 22513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3: Bootstrap starting.
I20260620 19:12:08.094058 22513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:08.094291 22513 log.cc:824] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:08.094990 22513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3: No bootstrap required, opened a new log
I20260620 19:12:08.096624 22513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d525d697c374f7bb8250499af145ad3" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35849 } }
I20260620 19:12:08.096763 22513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:08.096786 22513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d525d697c374f7bb8250499af145ad3, State: Initialized, Role: FOLLOWER
I20260620 19:12:08.096900 22513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d525d697c374f7bb8250499af145ad3" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35849 } }
I20260620 19:12:08.096951 22513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:08.096972 22513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:08.097004 22513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:08.097513 22513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d525d697c374f7bb8250499af145ad3" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35849 } }
I20260620 19:12:08.097597 22513 leader_election.cc:302] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d525d697c374f7bb8250499af145ad3; no voters:
I20260620 19:12:08.097728 22513 leader_election.cc:288] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:08.097800 22516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:08.097988 22516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3 [term 1 LEADER]: Becoming Leader. State: Replica: 2d525d697c374f7bb8250499af145ad3, State: Running, Role: LEADER
I20260620 19:12:08.097999 22513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:08.098093 22516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d525d697c374f7bb8250499af145ad3" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35849 } }
I20260620 19:12:08.098358 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 22445
I20260620 19:12:08.098481 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:12:08.098670 812 external_mini_cluster.cc:1428] Setting key 492647c76299a2d9ba5efe4352fa61c7
I20260620 19:12:08.099370 22518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d525d697c374f7bb8250499af145ad3. Latest consensus state: current_term: 1 leader_uuid: "2d525d697c374f7bb8250499af145ad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d525d697c374f7bb8250499af145ad3" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35849 } } }
I20260620 19:12:08.099443 22517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d525d697c374f7bb8250499af145ad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d525d697c374f7bb8250499af145ad3" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 35849 } } }
I20260620 19:12:08.099829 22517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:08.099485 22518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3 [sys.catalog]: This master's current role is: LEADER
W20260620 19:12:08.100459 22532 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:12:08.100541 22532 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:12:08.101684 22533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:08.101879 22533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:08.103515 22533 catalog_manager.cc:1347] Generated new cluster ID: a58ce6d6d88d4139bebfc9821a84ec8e
I20260620 19:12:08.103569 22533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:08.104642 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:08.099240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52767 (local address 127.0.203.62:35849)
0620 19:12:08.099670 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:08.099679 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:08.099714 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:12:08.099846 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:08.099855 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:08.099916 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:08.100073 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:08.100629 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:08.101546 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:08.102477 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:08.102994 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:08.103145 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:08.103927 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:08.103931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:08.103931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:08.104141 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:08.104194 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:08.104208 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:08.104286 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:08.104419 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 19:12:08.104445 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":74,"threads_started":1}
I20260620 19:12:08.105563 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:12:08.110803 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:08.107060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42951 (local address 127.0.203.62:35849)
0620 19:12:08.107092 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:08.107093 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:08.107381 (+ 288us) server_negotiation.cc:407] Connection header received
0620 19:12:08.107437 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:08.107437 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:08.107448 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:08.107502 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:08.107637 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:08.108358 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:08.109451 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:08.109775 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:08.109842 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:08.110540 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:08.110542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:08.110542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:08.110546 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:08.110562 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:08.110566 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:08.110626 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:08.110709 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 19:12:08.110721 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:12:08.111997 22533 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:08.112687 22533 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:08.117280 22533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d525d697c374f7bb8250499af145ad3: Generated new TSK 0
I20260620 19:12:08.117489 22533 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:12:08.120028 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 22445
W20260620 19:12:08.125891 22430 connection.cc:535] client connection to 127.0.203.62:35849 recv error: Network error: failed to read from TLS socket (remote: 127.0.203.62:35849): Connection reset by peer (error 104)
2026-06-20T19:12:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:08 dist-test-slave-wd30 krb5kdc[22544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:08 dist-test-slave-wd30 krb5kdc[22544](info): set up 2 sockets
Jun 20 19:12:08 dist-test-slave-wd30 krb5kdc[22544](info): commencing operation
krb5kdc: starting...
W20260620 19:12:09.159349 812 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:12:09 dist-test-slave-wd30 krb5kdc[22544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:12:09.278122 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:37489
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:37489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:09.358894 22557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:09.359095 22557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:09.359133 22557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:09.360260 22557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:09.360296 22557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:09.360309 22557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:09.360322 22557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:09.360334 22557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:09.361712 22557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:37489
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:37489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:09.361898 22557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:09.362067 22557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:09.366973 22563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:09.367010 22565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:09.366981 22562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:09.367204 22557 server_base.cc:1034] running on GCE node
I20260620 19:12:09.367493 22557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:09.367777 22557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:09.369028 22557 hybrid_clock.cc:648] HybridClock initialized: now 1781982729369019 us; error 39 us; skew 500 ppm
Jun 20 19:12:09 dist-test-slave-wd30 krb5kdc[22544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:09.371675 22557 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:12:09.372144 22557 webserver.cc:460] Webserver started at http://127.0.203.62:45823/ using document root <none> and password file <none>
I20260620 19:12:09.372422 22557 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:09.372489 22557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:09.372624 22557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:09.373704 22557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "c840cdd75bb84bdbb1d71cb26db6f1c5"
format_stamp: "Formatted at 2026-06-20 19:12:09 on dist-test-slave-wd30"
server_key: "acbbdc13953cc1f502e686880d9ceff3"
server_key_iv: "d327783900f47ccb6be5126703dd01c9"
server_key_version: "encryptionkey@0"
I20260620 19:12:09.374073 22557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "c840cdd75bb84bdbb1d71cb26db6f1c5"
format_stamp: "Formatted at 2026-06-20 19:12:09 on dist-test-slave-wd30"
server_key: "acbbdc13953cc1f502e686880d9ceff3"
server_key_iv: "d327783900f47ccb6be5126703dd01c9"
server_key_version: "encryptionkey@0"
I20260620 19:12:09.375573 22557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:12:09.376428 22572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:09.376637 22557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:09.376747 22557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "c840cdd75bb84bdbb1d71cb26db6f1c5"
format_stamp: "Formatted at 2026-06-20 19:12:09 on dist-test-slave-wd30"
server_key: "acbbdc13953cc1f502e686880d9ceff3"
server_key_iv: "d327783900f47ccb6be5126703dd01c9"
server_key_version: "encryptionkey@0"
I20260620 19:12:09.376832 22557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:09.380183 22557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:09.382023 22557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:09.382176 22557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:09.386209 22557 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:37489
I20260620 19:12:09.386255 22624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:37489 every 8 connection(s)
I20260620 19:12:09.386588 22557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:09.387609 22625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:09.389837 22625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5: Bootstrap starting.
I20260620 19:12:09.390414 22625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:09.390697 22625 log.cc:824] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:09.391427 22625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5: No bootstrap required, opened a new log
I20260620 19:12:09.392735 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 22557
I20260620 19:12:09.392829 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:12:09.393018 812 external_mini_cluster.cc:1428] Setting key 8691f639bf16ebdf28ccaca227b6c5d9
I20260620 19:12:09.393107 22625 raft_consensus.cc:348] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c840cdd75bb84bdbb1d71cb26db6f1c5" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 37489 } }
I20260620 19:12:09.393319 22625 raft_consensus.cc:374] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:09.393365 22625 raft_consensus.cc:729] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c840cdd75bb84bdbb1d71cb26db6f1c5, State: Initialized, Role: FOLLOWER
I20260620 19:12:09.393534 22625 consensus_queue.cc:260] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c840cdd75bb84bdbb1d71cb26db6f1c5" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 37489 } }
I20260620 19:12:09.393643 22625 raft_consensus.cc:388] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:09.393702 22625 raft_consensus.cc:482] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:09.393764 22625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:09.394863 22625 raft_consensus.cc:504] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c840cdd75bb84bdbb1d71cb26db6f1c5" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 37489 } }
I20260620 19:12:09.394991 22625 leader_election.cc:302] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c840cdd75bb84bdbb1d71cb26db6f1c5; no voters:
I20260620 19:12:09.395175 22625 leader_election.cc:288] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:09.395211 22630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:09.395403 22630 raft_consensus.cc:686] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5 [term 1 LEADER]: Becoming Leader. State: Replica: c840cdd75bb84bdbb1d71cb26db6f1c5, State: Running, Role: LEADER
I20260620 19:12:09.395442 22625 sys_catalog.cc:564] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:09.395555 22630 consensus_queue.cc:237] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c840cdd75bb84bdbb1d71cb26db6f1c5" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 37489 } }
I20260620 19:12:09.395828 22629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:09.393557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36139 (local address 127.0.203.62:37489)
0620 19:12:09.393792 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:09.393801 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:09.393834 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:12:09.393967 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:09.393974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:09.394004 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:09.394183 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:09.395219 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:09.395222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:09.395223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:09.395228 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:09.395274 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:09.395290 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:09.395530 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:09.395561 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 19:12:09.395602 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":94,"threads_started":1}
I20260620 19:12:09.395922 22631 sys_catalog.cc:455] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c840cdd75bb84bdbb1d71cb26db6f1c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c840cdd75bb84bdbb1d71cb26db6f1c5" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 37489 } } }
I20260620 19:12:09.396023 22631 sys_catalog.cc:458] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:09.395918 22632 sys_catalog.cc:455] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c840cdd75bb84bdbb1d71cb26db6f1c5. Latest consensus state: current_term: 1 leader_uuid: "c840cdd75bb84bdbb1d71cb26db6f1c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c840cdd75bb84bdbb1d71cb26db6f1c5" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 37489 } } }
I20260620 19:12:09.396447 22632 sys_catalog.cc:458] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:09.396450 22636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:09.397028 22636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:09.397341 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:12:09.399837 22629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:09.398709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41353 (local address 127.0.203.62:37489)
0620 19:12:09.398740 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:09.398742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:09.398873 (+ 131us) server_negotiation.cc:407] Connection header received
0620 19:12:09.398942 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:09.398943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:09.398947 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:09.398990 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:09.399602 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:09.399603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:09.399603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:09.399607 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:09.399622 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:09.399626 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:09.399688 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:09.399764 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 19:12:09.399778 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:12:09.399847 22636 catalog_manager.cc:1347] Generated new cluster ID: e8b81587bb044d67b11d1b5fce97f189
I20260620 19:12:09.399940 22636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:09.422690 22636 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:09.422917 22636 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:09.427728 22636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c840cdd75bb84bdbb1d71cb26db6f1c5: Generated new TSK 0
I20260620 19:12:09.427937 22636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:12:09.437721 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 22557
2026-06-20T19:12:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:09 dist-test-slave-wd30 krb5kdc[22656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:09 dist-test-slave-wd30 krb5kdc[22656](info): set up 2 sockets
Jun 20 19:12:09 dist-test-slave-wd30 krb5kdc[22656](info): commencing operation
krb5kdc: starting...
W20260620 19:12:10.475279 812 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:12:10 dist-test-slave-wd30 krb5kdc[22656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:12:10.600292 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:42499
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:39725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:42499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:10.681914 22669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:10.682142 22669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:10.682168 22669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:10.683331 22669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:10.683368 22669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:10.683382 22669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:10.683395 22669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:10.683408 22669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:10.684831 22669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:39725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:42499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:42499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:10.685040 22669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:10.685226 22669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:12:10.690335 22669 server_base.cc:1034] running on GCE node
W20260620 19:12:10.690313 22674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:10.690316 22677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:10.690342 22675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:10.690891 22669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:10.691262 22669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:10.692394 22669 hybrid_clock.cc:648] HybridClock initialized: now 1781982730692382 us; error 30 us; skew 500 ppm
Jun 20 19:12:10 dist-test-slave-wd30 krb5kdc[22656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:10.694998 22669 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:12:10.695472 22669 webserver.cc:460] Webserver started at http://127.0.203.62:46445/ using document root <none> and password file <none>
I20260620 19:12:10.695678 22669 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:10.695719 22669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:10.695806 22669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:10.696849 22669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "08c92007a6d340a0ade488051ed5e789"
format_stamp: "Formatted at 2026-06-20 19:12:10 on dist-test-slave-wd30"
server_key: "4c6b8c27751da838f472d63acff0a42a"
server_key_iv: "7e2317c69c5a48381ad53a20f86c8001"
server_key_version: "encryptionkey@0"
I20260620 19:12:10.697166 22669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "08c92007a6d340a0ade488051ed5e789"
format_stamp: "Formatted at 2026-06-20 19:12:10 on dist-test-slave-wd30"
server_key: "4c6b8c27751da838f472d63acff0a42a"
server_key_iv: "7e2317c69c5a48381ad53a20f86c8001"
server_key_version: "encryptionkey@0"
I20260620 19:12:10.698525 22669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:12:10.699213 22684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:10.699412 22669 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 19:12:10.699513 22669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "08c92007a6d340a0ade488051ed5e789"
format_stamp: "Formatted at 2026-06-20 19:12:10 on dist-test-slave-wd30"
server_key: "4c6b8c27751da838f472d63acff0a42a"
server_key_iv: "7e2317c69c5a48381ad53a20f86c8001"
server_key_version: "encryptionkey@0"
I20260620 19:12:10.699604 22669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:10.715739 22669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:10.718369 22669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:10.718577 22669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:10.722611 22669 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:42499
I20260620 19:12:10.722648 22736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:42499 every 8 connection(s)
I20260620 19:12:10.723022 22669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:10.723976 22737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:10.725188 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 22669
I20260620 19:12:10.725299 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:12:10.725490 812 external_mini_cluster.cc:1428] Setting key 6641a60d5f378212de58fc10e5da8e00
I20260620 19:12:10.726979 22737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789: Bootstrap starting.
I20260620 19:12:10.727609 22737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:10.727897 22737 log.cc:824] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:10.728571 22737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789: No bootstrap required, opened a new log
I20260620 19:12:10.730293 22737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c92007a6d340a0ade488051ed5e789" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42499 } }
I20260620 19:12:10.730430 22737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:10.730453 22737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c92007a6d340a0ade488051ed5e789, State: Initialized, Role: FOLLOWER
I20260620 19:12:10.730603 22737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c92007a6d340a0ade488051ed5e789" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42499 } }
I20260620 19:12:10.730679 22737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:10.730716 22737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:10.730764 22737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:10.731383 22737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c92007a6d340a0ade488051ed5e789" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42499 } }
I20260620 19:12:10.731470 22737 leader_election.cc:302] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08c92007a6d340a0ade488051ed5e789; no voters:
I20260620 19:12:10.731688 22737 leader_election.cc:288] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:10.731710 22742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:10.731829 22742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789 [term 1 LEADER]: Becoming Leader. State: Replica: 08c92007a6d340a0ade488051ed5e789, State: Running, Role: LEADER
I20260620 19:12:10.731900 22742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c92007a6d340a0ade488051ed5e789" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42499 } }
I20260620 19:12:10.731959 22737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:10.732223 22742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08c92007a6d340a0ade488051ed5e789. Latest consensus state: current_term: 1 leader_uuid: "08c92007a6d340a0ade488051ed5e789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c92007a6d340a0ade488051ed5e789" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42499 } } }
I20260620 19:12:10.732281 22742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:10.732240 22743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08c92007a6d340a0ade488051ed5e789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c92007a6d340a0ade488051ed5e789" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 42499 } } }
I20260620 19:12:10.732323 22743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789 [sys.catalog]: This master's current role is: LEADER
Jun 20 19:12:10 dist-test-slave-wd30 krb5kdc[22656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:12:10.732470 22746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:10.733070 22746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:10.736291 22746 catalog_manager.cc:1347] Generated new cluster ID: 9eb9da40665a436397b4314b8164deef
I20260620 19:12:10.736347 22746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:10.737164 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:10.726032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46305 (local address 127.0.203.62:42499)
0620 19:12:10.726228 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:10.726239 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:10.726274 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:12:10.726801 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:10.726810 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:10.726882 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:10.727039 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:10.727782 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:10.728801 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:10.729573 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:10.729987 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:10.730106 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:10.732917 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:10.732921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:10.732921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:10.732935 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:10.735064 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:10.735536 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:10.735539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:10.735539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:10.735666 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:10.735923 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:10.735924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:10.735925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:10.736214 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:10.736746 (+ 532us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:10.736815 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 19:12:10.736949 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260620 19:12:10.738117 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:12:10.743845 22746 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:10.744477 22746 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:10.747107 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:10.739498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58697 (local address 127.0.203.62:42499)
0620 19:12:10.739562 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:10.739564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:10.739747 (+ 183us) server_negotiation.cc:407] Connection header received
0620 19:12:10.739778 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:10.739779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:10.739790 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:10.739859 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:10.740456 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:10.741286 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:10.742280 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:10.742632 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:10.742709 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:10.744203 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:10.744204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:10.744205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:10.744213 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:10.745957 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:10.746335 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:10.746337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:10.746337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:10.746400 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:10.746582 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:10.746582 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:10.746583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:10.746711 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:10.746808 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:10.746910 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 19:12:10.747011 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:12:10.750017 22746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08c92007a6d340a0ade488051ed5e789: Generated new TSK 0
I20260620 19:12:10.750216 22746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:12:10.752671 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 22669
2026-06-20T19:12:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:10 dist-test-slave-wd30 krb5kdc[22768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:10 dist-test-slave-wd30 krb5kdc[22768](info): set up 2 sockets
Jun 20 19:12:10 dist-test-slave-wd30 krb5kdc[22768](info): commencing operation
krb5kdc: starting...
W20260620 19:12:11.787333 812 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:12:11 dist-test-slave-wd30 krb5kdc[22768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:12:11.902416 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:41399
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:36777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:41399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:11.982173 22781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:11.982415 22781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:11.982455 22781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:11.983637 22781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:11.983711 22781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:11.983737 22781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:11.983767 22781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:11.983796 22781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:11.985229 22781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:36777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:41399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:41399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:11.985496 22781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:11.985723 22781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:11.990386 22787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:11.990393 22789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:11.990393 22786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:11.990597 22781 server_base.cc:1034] running on GCE node
I20260620 19:12:11.990921 22781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:11.991243 22781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:11.992406 22781 hybrid_clock.cc:648] HybridClock initialized: now 1781982731992400 us; error 34 us; skew 500 ppm
Jun 20 19:12:11 dist-test-slave-wd30 krb5kdc[22768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:11.998070 22781 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:12:11.998504 22781 webserver.cc:460] Webserver started at http://127.0.203.62:42077/ using document root <none> and password file <none>
I20260620 19:12:11.998730 22781 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:11.998799 22781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:11.998950 22781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:12.000082 22781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "3f86cd5cd468499286dfad3938c7d727"
format_stamp: "Formatted at 2026-06-20 19:12:11 on dist-test-slave-wd30"
server_key: "17ce6b3bcf2dc020f0880cffde23615c"
server_key_iv: "831a624afc01b86d5ef50bdf831b664a"
server_key_version: "encryptionkey@0"
I20260620 19:12:12.000459 22781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "3f86cd5cd468499286dfad3938c7d727"
format_stamp: "Formatted at 2026-06-20 19:12:11 on dist-test-slave-wd30"
server_key: "17ce6b3bcf2dc020f0880cffde23615c"
server_key_iv: "831a624afc01b86d5ef50bdf831b664a"
server_key_version: "encryptionkey@0"
I20260620 19:12:12.001807 22781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:12:12.003394 22796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:12.003562 22781 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:12.003664 22781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "3f86cd5cd468499286dfad3938c7d727"
format_stamp: "Formatted at 2026-06-20 19:12:11 on dist-test-slave-wd30"
server_key: "17ce6b3bcf2dc020f0880cffde23615c"
server_key_iv: "831a624afc01b86d5ef50bdf831b664a"
server_key_version: "encryptionkey@0"
I20260620 19:12:12.003721 22781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:12.015694 22781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:12.017485 22781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:12.017598 22781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:12.021456 22781 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:41399
I20260620 19:12:12.021520 22848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:41399 every 8 connection(s)
I20260620 19:12:12.021872 22781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:12.022730 22849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:12.025034 22849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727: Bootstrap starting.
I20260620 19:12:12.025607 22849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:12.025913 22849 log.cc:824] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:12.026616 22849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727: No bootstrap required, opened a new log
I20260620 19:12:12.026923 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 22781
I20260620 19:12:12.027043 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:12:12.027271 812 external_mini_cluster.cc:1428] Setting key 3de44111e507ea0adaa226d5f4094b76
I20260620 19:12:12.028769 22849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f86cd5cd468499286dfad3938c7d727" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41399 } }
I20260620 19:12:12.028931 22849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:12.028970 22849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f86cd5cd468499286dfad3938c7d727, State: Initialized, Role: FOLLOWER
I20260620 19:12:12.029150 22849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f86cd5cd468499286dfad3938c7d727" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41399 } }
I20260620 19:12:12.029258 22849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:12.029304 22849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:12.029354 22849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:12.029879 22849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f86cd5cd468499286dfad3938c7d727" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41399 } }
I20260620 19:12:12.029994 22849 leader_election.cc:302] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f86cd5cd468499286dfad3938c7d727; no voters:
I20260620 19:12:12.030195 22849 leader_election.cc:288] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:12.030227 22854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:12.030330 22854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727 [term 1 LEADER]: Becoming Leader. State: Replica: 3f86cd5cd468499286dfad3938c7d727, State: Running, Role: LEADER
I20260620 19:12:12.030419 22854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f86cd5cd468499286dfad3938c7d727" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41399 } }
I20260620 19:12:12.030452 22849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:12.030917 22854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f86cd5cd468499286dfad3938c7d727. Latest consensus state: current_term: 1 leader_uuid: "3f86cd5cd468499286dfad3938c7d727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f86cd5cd468499286dfad3938c7d727" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41399 } } }
I20260620 19:12:12.030926 22856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f86cd5cd468499286dfad3938c7d727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f86cd5cd468499286dfad3938c7d727" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41399 } } }
I20260620 19:12:12.031075 22856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:12.031306 22854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:12.031755 22860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:12.032254 22860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:12.033751 22860 catalog_manager.cc:1347] Generated new cluster ID: a77feed1e4784c9faf19d03d71f162da
I20260620 19:12:12.033819 22860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:12.034405 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:12.027781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44179 (local address 127.0.203.62:41399)
0620 19:12:12.027963 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:12.027972 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:12.028009 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:12:12.028751 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:12.028757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:12.028804 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:12.028912 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:12.029179 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:12.030858 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:12.032126 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:12.032750 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:12.032878 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:12.033595 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:12.033599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:12.033599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:12.033841 (+ 242us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:12.033881 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:12.033896 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:12.033963 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:12.034053 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 19:12:12.034197 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260620 19:12:12.035104 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:12:12.041419 22860 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:12.042033 22860 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:12.042520 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:12.036496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38993 (local address 127.0.203.62:41399)
0620 19:12:12.036527 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:12.036528 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:12.039293 (+ 2765us) server_negotiation.cc:407] Connection header received
0620 19:12:12.039353 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:12.039353 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:12.039363 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:12.039420 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:12.039569 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:12.040279 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:12.041243 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:12.041516 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:12.041584 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:12.042273 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:12.042274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:12.042274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:12.042279 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:12.042292 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:12.042296 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:12.042349 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:12.042414 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 19:12:12.042451 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:12:12.048029 22860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f86cd5cd468499286dfad3938c7d727: Generated new TSK 0
I20260620 19:12:12.048208 22860 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:12:12.053432 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 22781
2026-06-20T19:12:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:12 dist-test-slave-wd30 krb5kdc[22880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:12 dist-test-slave-wd30 krb5kdc[22880](info): set up 2 sockets
Jun 20 19:12:12 dist-test-slave-wd30 krb5kdc[22880](info): commencing operation
krb5kdc: starting...
W20260620 19:12:13.083432 812 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:12:13 dist-test-slave-wd30 krb5kdc[22880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:12:13.223871 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:36855
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:35235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:36855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:13.308394 22893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:13.308630 22893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:13.308656 22893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:13.309908 22893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:13.309957 22893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:13.309973 22893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:13.309985 22893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:13.309998 22893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:13.311645 22893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:35235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:36855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:36855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:13.311915 22893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:13.312160 22893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:13.317500 22898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:13.317533 22901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:13.317574 22899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:13.317886 22893 server_base.cc:1034] running on GCE node
I20260620 19:12:13.318192 22893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:13.318532 22893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:13.319725 22893 hybrid_clock.cc:648] HybridClock initialized: now 1781982733319697 us; error 47 us; skew 500 ppm
Jun 20 19:12:13 dist-test-slave-wd30 krb5kdc[22880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:13.322451 22893 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:12:13.322963 22893 webserver.cc:460] Webserver started at http://127.0.203.62:44127/ using document root <none> and password file <none>
I20260620 19:12:13.323303 22893 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:13.323382 22893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:13.323541 22893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:13.324757 22893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "26dccce8c3da40408c000fbb1aa8e8cd"
format_stamp: "Formatted at 2026-06-20 19:12:13 on dist-test-slave-wd30"
server_key: "99f572687baddd685a51c639e46d20ca"
server_key_iv: "36104eb7a34999d850d78319acc90b23"
server_key_version: "encryptionkey@0"
I20260620 19:12:13.325186 22893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "26dccce8c3da40408c000fbb1aa8e8cd"
format_stamp: "Formatted at 2026-06-20 19:12:13 on dist-test-slave-wd30"
server_key: "99f572687baddd685a51c639e46d20ca"
server_key_iv: "36104eb7a34999d850d78319acc90b23"
server_key_version: "encryptionkey@0"
I20260620 19:12:13.326758 22893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:12:13.327692 22908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:13.327903 22893 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 19:12:13.328015 22893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "26dccce8c3da40408c000fbb1aa8e8cd"
format_stamp: "Formatted at 2026-06-20 19:12:13 on dist-test-slave-wd30"
server_key: "99f572687baddd685a51c639e46d20ca"
server_key_iv: "36104eb7a34999d850d78319acc90b23"
server_key_version: "encryptionkey@0"
I20260620 19:12:13.328097 22893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:13.331784 22893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:13.333811 22893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:13.333976 22893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:13.337982 22893 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:36855
I20260620 19:12:13.338047 22960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:36855 every 8 connection(s)
I20260620 19:12:13.338394 22893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:13.339458 22961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:13.339653 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 22893
I20260620 19:12:13.339748 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:12:13.339931 812 external_mini_cluster.cc:1428] Setting key b3df58425187f742707bec13ce470ae0
I20260620 19:12:13.342252 22964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:13.340416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57129 (local address 127.0.203.62:36855)
0620 19:12:13.340577 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:13.340584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:13.340613 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:12:13.341056 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:13.341062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:13.341087 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:13.341198 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:13.341809 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:13.341812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:13.341812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:13.341816 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:13.341857 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:13.341867 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:13.341919 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:13.342003 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 19:12:13.342058 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260620 19:12:13.343024 22961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd: Bootstrap starting.
I20260620 19:12:13.343178 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:12:13.343773 22961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:13.344120 22961 log.cc:824] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:13.344940 22961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd: No bootstrap required, opened a new log
I20260620 19:12:13.345852 22964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:13.344794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58599 (local address 127.0.203.62:36855)
0620 19:12:13.344850 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:13.344851 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:13.344921 (+ 70us) server_negotiation.cc:407] Connection header received
0620 19:12:13.345003 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:13.345004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:13.345009 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:13.345068 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:13.345638 (+ 570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:13.345639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:13.345640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:13.345644 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:13.345656 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:13.345660 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:13.345715 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:13.345771 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 19:12:13.345784 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:12:13.346920 22961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26dccce8c3da40408c000fbb1aa8e8cd" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36855 } }
I20260620 19:12:13.347052 22961 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:13.347074 22961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26dccce8c3da40408c000fbb1aa8e8cd, State: Initialized, Role: FOLLOWER
I20260620 19:12:13.347235 22961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26dccce8c3da40408c000fbb1aa8e8cd" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36855 } }
I20260620 19:12:13.347307 22961 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:13.347362 22961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:13.347414 22961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:13.348174 22961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26dccce8c3da40408c000fbb1aa8e8cd" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36855 } }
I20260620 19:12:13.348311 22961 leader_election.cc:302] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26dccce8c3da40408c000fbb1aa8e8cd; no voters:
I20260620 19:12:13.348496 22961 leader_election.cc:288] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:13.348523 22971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:13.348668 22971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd [term 1 LEADER]: Becoming Leader. State: Replica: 26dccce8c3da40408c000fbb1aa8e8cd, State: Running, Role: LEADER
I20260620 19:12:13.348739 22961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:13.348753 22971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26dccce8c3da40408c000fbb1aa8e8cd" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36855 } }
I20260620 19:12:13.349112 22972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26dccce8c3da40408c000fbb1aa8e8cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26dccce8c3da40408c000fbb1aa8e8cd" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36855 } } }
I20260620 19:12:13.349115 22973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26dccce8c3da40408c000fbb1aa8e8cd. Latest consensus state: current_term: 1 leader_uuid: "26dccce8c3da40408c000fbb1aa8e8cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26dccce8c3da40408c000fbb1aa8e8cd" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 36855 } } }
I20260620 19:12:13.349231 22972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:13.349239 22973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd [sys.catalog]: This master's current role is: LEADER
W20260620 19:12:13.350517 22987 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:12:13.350579 22987 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:12:13.350667 22977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:13.350867 22977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:13.352277 22977 catalog_manager.cc:1347] Generated new cluster ID: 4e641763041149ae823cb6963ca1d923
I20260620 19:12:13.352331 22977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:13.374855 22977 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:13.375073 22977 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:13.381594 22977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26dccce8c3da40408c000fbb1aa8e8cd: Generated new TSK 0
I20260620 19:12:13.381836 22977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:12:13.385828 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 22893
2026-06-20T19:12:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1332 ms)
[----------] 18 tests from AuthTokenIssuingTest (23776 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:13 dist-test-slave-wd30 krb5kdc[22992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:13 dist-test-slave-wd30 krb5kdc[22992](info): set up 2 sockets
Jun 20 19:12:13 dist-test-slave-wd30 krb5kdc[22992](info): commencing operation
krb5kdc: starting...
W20260620 19:12:14.423442 812 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:12:14 dist-test-slave-wd30 krb5kdc[22992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:12:14.544334 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:41907
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:36485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:41907,127.0.203.61:34143,127.0.203.60:34679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:14.626241 23005 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:12:14.626541 23005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:14.626575 23005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:14.626628 23005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:14.627918 23005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:14.627995 23005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:14.628018 23005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:14.628041 23005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:14.628072 23005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:14.629673 23005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:36485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:41907,127.0.203.61:34143,127.0.203.60:34679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:41907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:14.629963 23005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:14.630210 23005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:14.635301 23011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:14.635311 23010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:14.635300 23013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:14.635520 23005 server_base.cc:1034] running on GCE node
I20260620 19:12:14.635764 23005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:14.636082 23005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:14.637251 23005 hybrid_clock.cc:648] HybridClock initialized: now 1781982734637239 us; error 44 us; skew 500 ppm
Jun 20 19:12:14 dist-test-slave-wd30 krb5kdc[22992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:14.639736 23005 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:12:14.640122 23005 webserver.cc:460] Webserver started at http://127.0.203.62:44515/ using document root <none> and password file <none>
I20260620 19:12:14.640298 23005 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:14.640336 23005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:14.640419 23005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:14.641415 23005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "b47b27b19a484c5bac5f55faec750367"
format_stamp: "Formatted at 2026-06-20 19:12:14 on dist-test-slave-wd30"
server_key: "f5eb6b8a824e1957b024f5c462de874e"
server_key_iv: "5c037666143ed65222714e4697d230b0"
server_key_version: "encryptionkey@0"
I20260620 19:12:14.641713 23005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "b47b27b19a484c5bac5f55faec750367"
format_stamp: "Formatted at 2026-06-20 19:12:14 on dist-test-slave-wd30"
server_key: "f5eb6b8a824e1957b024f5c462de874e"
server_key_iv: "5c037666143ed65222714e4697d230b0"
server_key_version: "encryptionkey@0"
I20260620 19:12:14.642922 23005 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:12:14.643708 23020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:14.643857 23005 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:14.643942 23005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "b47b27b19a484c5bac5f55faec750367"
format_stamp: "Formatted at 2026-06-20 19:12:14 on dist-test-slave-wd30"
server_key: "f5eb6b8a824e1957b024f5c462de874e"
server_key_iv: "5c037666143ed65222714e4697d230b0"
server_key_version: "encryptionkey@0"
I20260620 19:12:14.643996 23005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:14.652381 23005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:14.654021 23005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:14.654120 23005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:14.657742 23005 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:41907
I20260620 19:12:14.657789 23072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:41907 every 8 connection(s)
I20260620 19:12:14.658095 23005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:14.658753 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 23005
I20260620 19:12:14.658829 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:12:14.659016 812 external_mini_cluster.cc:1428] Setting key dfc141a0a864337d9a0edfee48f4ad64
I20260620 19:12:14.659229 23073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:14.660722 23073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.203.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.61@KRBTEST.COM" created.
Jun 20 19:12:14 dist-test-slave-wd30 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:12:14.673614 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.661509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:41907 (local address 127.0.0.1:52736)
0620 19:12:14.661705 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.661744 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:14.662470 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:14.662812 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:14.662814 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:14.663243 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:14.663482 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:14.663488 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.664544 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.664545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:14.665281 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:14.665283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.665820 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.665821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:14.665886 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:14.666513 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:14.666529 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:14.670091 (+ 3562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:14.671987 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:14.671993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:14.671996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:14.672204 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:14.672469 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:14.672470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:14.672471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:14.672557 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:14.672933 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:14.672938 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:14.673012 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 19:12:14.673137 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 19:12:14.673391 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":96,"thread_start_us":81,"threads_started":1}
I20260620 19:12:14.673612 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.661548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.0.203.62:41907)
0620 19:12:14.662613 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:14.662616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:14.662627 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:12:14.662684 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:14.662685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:14.662701 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:14.662791 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:14.663602 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.664431 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.665382 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.665703 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.665832 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:14.670197 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:14.670205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:14.670205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:14.670212 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:14.671892 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:14.672306 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:14.672309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:14.672310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:14.672398 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:14.672624 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:14.672626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:14.672626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:14.672859 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:14.672957 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:14.673170 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 19:12:14.673344 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":1044,"thread_start_us":122,"threads_started":1}
I20260620 19:12:14.674347 23073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:12:14.674767 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.674614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:34143 (local address 127.0.0.1:47370)
0620 19:12:14.674673 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.674733 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.61:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 19:12:14.674929 23023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.203.61:34143: connect: Connection refused (error 111)
I20260620 19:12:14.675446 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.675336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:34143 (local address 127.0.0.1:47386)
0620 19:12:14.675388 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.675419 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.61:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 19:12:14.675573 23073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.61:34143: Network error: Client connection negotiation failed: client connection to 127.0.203.61:34143: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.61.keytab.
Entry for principal kudu/127.0.203.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.61.keytab.
I20260620 19:12:14.686093 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.61:34143
--webserver_interface=127.0.203.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.61.keytab
--principal=kudu/127.0.203.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:36485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:41907,127.0.203.61:34143,127.0.203.60:34679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:12:14.724831 23073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } attempt: 1
I20260620 19:12:14.725407 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.725239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:34143 (local address 127.0.0.1:47388)
0620 19:12:14.725311 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.725365 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.61:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 19:12:14.725798 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.725701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:34143 (local address 127.0.0.1:47390)
0620 19:12:14.725747 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.725776 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.61:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 19:12:14.725975 23073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.61:34143: Network error: Client connection negotiation failed: client connection to 127.0.203.61:34143: connect: Connection refused (error 111)
W20260620 19:12:14.771528 23080 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:12:14.771761 23080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:14.771780 23080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:14.771829 23080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:14.773088 23080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:14.773133 23080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:14.773149 23080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:14.773161 23080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:14.773212 23080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:14.774766 23080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:36485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:41907,127.0.203.61:34143,127.0.203.60:34679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.61.keytab
--principal=kudu/127.0.203.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.61:34143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.203.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:14.775040 23080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:14.775324 23080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:14.780434 23085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:14.780436 23088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:14.780453 23086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:14.780737 23080 server_base.cc:1034] running on GCE node
I20260620 19:12:14.781023 23080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:14.781347 23080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:14.782509 23080 hybrid_clock.cc:648] HybridClock initialized: now 1781982734782497 us; error 29 us; skew 500 ppm
Jun 20 19:12:14 dist-test-slave-wd30 krb5kdc[22992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:14.785306 23080 init.cc:377] Logged in from keytab as kudu/127.0.203.61@KRBTEST.COM (short username kudu)
I20260620 19:12:14.785912 23080 webserver.cc:460] Webserver started at http://127.0.203.61:33961/ using document root <none> and password file <none>
I20260620 19:12:14.786171 23080 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:14.786242 23080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:14.786397 23080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:14.787607 23080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-1/data/instance:
uuid: "a8cdf5225b614c3db6bae4db85edb728"
format_stamp: "Formatted at 2026-06-20 19:12:14 on dist-test-slave-wd30"
server_key: "54f9e55475a7f568522d8f95cb07785e"
server_key_iv: "31da3f7ed06ac0dc9d5f503a6cbcbf30"
server_key_version: "encryptionkey@0"
I20260620 19:12:14.787994 23080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-1/wal/instance:
uuid: "a8cdf5225b614c3db6bae4db85edb728"
format_stamp: "Formatted at 2026-06-20 19:12:14 on dist-test-slave-wd30"
server_key: "54f9e55475a7f568522d8f95cb07785e"
server_key_iv: "31da3f7ed06ac0dc9d5f503a6cbcbf30"
server_key_version: "encryptionkey@0"
I20260620 19:12:14.789506 23080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:12:14.790290 23095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:14.790544 23080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:14.790696 23080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-1/wal
uuid: "a8cdf5225b614c3db6bae4db85edb728"
format_stamp: "Formatted at 2026-06-20 19:12:14 on dist-test-slave-wd30"
server_key: "54f9e55475a7f568522d8f95cb07785e"
server_key_iv: "31da3f7ed06ac0dc9d5f503a6cbcbf30"
server_key_version: "encryptionkey@0"
I20260620 19:12:14.790794 23080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:14.794186 23073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } attempt: 2
I20260620 19:12:14.794657 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.794508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:34143 (local address 127.0.0.1:47406)
0620 19:12:14.794575 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.794621 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.61:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 19:12:14.795035 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.794945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:34143 (local address 127.0.0.1:47408)
0620 19:12:14.794990 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.795016 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.61:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 19:12:14.795202 23073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.61:34143: Network error: Client connection negotiation failed: client connection to 127.0.203.61:34143: connect: Connection refused (error 111)
I20260620 19:12:14.802477 23080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:14.804869 23080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:14.805051 23080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:14.809538 23080 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.61:34143
I20260620 19:12:14.809576 23147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.61:34143 every 8 connection(s)
I20260620 19:12:14.809947 23080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-1/data/info.pb
I20260620 19:12:14.810961 23148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:14.811414 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 23080
I20260620 19:12:14.811522 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-1/wal/instance
I20260620 19:12:14.811746 812 external_mini_cluster.cc:1428] Setting key 7ed3cf7e5f8ddf427807a5bfe12d5274
I20260620 19:12:14.813513 23148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.203.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.60@KRBTEST.COM" created.
Jun 20 19:12:14 dist-test-slave-wd30 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.61@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:12:14.826891 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.814326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.0.203.62:41907)
0620 19:12:14.814404 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:14.814406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:14.814573 (+ 167us) server_negotiation.cc:407] Connection header received
0620 19:12:14.815067 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:14.815069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:14.815086 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:14.815211 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:14.815957 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.816800 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.817837 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.818175 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.818260 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:14.823378 (+ 5118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:14.823380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:14.823381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:14.823392 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:14.825149 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:14.825616 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:14.825617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:14.825618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:14.825688 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:14.825993 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:14.825994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:14.825994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:14.826159 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:14.826238 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:14.826670 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 19:12:14.826777 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:12:14.826992 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.814242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:41907 (local address 127.0.0.1:52748)
0620 19:12:14.814431 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.814475 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:14.814956 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:14.815295 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:14.815299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:14.815640 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:14.815848 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:14.815852 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.816908 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.816909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:14.817730 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:14.817732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.818287 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.818288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:14.818395 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:14.819012 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:14.819020 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:14.823254 (+ 4234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:14.825257 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:14.825266 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:14.825269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:14.825528 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:14.825782 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:14.825783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:14.825784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:14.825918 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:14.826272 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:14.826279 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:14.826474 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 19:12:14.826617 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 19:12:14.826772 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260620 19:12:14.827787 23148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:12:14 dist-test-slave-wd30 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.61@KRBTEST.COM for kudu/127.0.203.61@KRBTEST.COM
I20260620 19:12:14.836848 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.828114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:34143 (local address 127.0.0.1:47416)
0620 19:12:14.828180 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.828194 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:14.828244 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:14.828481 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:14.828482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:14.828664 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:14.828735 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:14.828737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.829590 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.829591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:14.830475 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:14.830478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.830991 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.830993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:14.831083 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:14.831672 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:14.831677 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:14.833854 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:14.835813 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:14.835813 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:14.835814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:14.835973 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:14.836216 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:14.836217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:14.836217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:14.836268 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:14.836601 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:14.836601 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:14.836644 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 19:12:14.836701 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 19:12:14.836780 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 19:12:14.836979 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.828156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47416 (local address 127.0.203.61:34143)
0620 19:12:14.828328 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:14.828333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:14.828355 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:12:14.828381 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:14.828382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:14.828393 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:14.828456 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:14.828810 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.829494 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.830596 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.830910 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.830963 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:14.833937 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:14.833945 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:14.833945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:14.833952 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:14.835739 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:14.836074 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:14.836076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:14.836077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:14.836143 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:14.836318 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:14.836319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:14.836319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:14.836456 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:14.836637 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:14.836749 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 19:12:14.836830 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":143,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.60.keytab.
Entry for principal kudu/127.0.203.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.60.keytab.
I20260620 19:12:14.837569 23148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:12:14.837921 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.837816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:34679 (local address 127.0.0.1:37386)
0620 19:12:14.837863 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.837905 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 19:12:14.838023 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.60:34679
--webserver_interface=127.0.203.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.60.keytab
--principal=kudu/127.0.203.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:36485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:41907,127.0.203.61:34143,127.0.203.60:34679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:14.838063 23099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.203.60:34679: connect: Connection refused (error 111)
I20260620 19:12:14.838449 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.838360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:34679 (local address 127.0.0.1:37398)
0620 19:12:14.838405 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.838428 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 19:12:14.838567 23148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.60:34679: Network error: Client connection negotiation failed: client connection to 127.0.203.60:34679: connect: Connection refused (error 111)
I20260620 19:12:14.886364 23073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } attempt: 3
I20260620 19:12:14.887801 23148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } attempt: 1
I20260620 19:12:14.888217 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.888067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:34679 (local address 127.0.0.1:37410)
0620 19:12:14.888131 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.888176 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 19:12:14.888564 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.888454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:34679 (local address 127.0.0.1:37422)
0620 19:12:14.888506 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.888543 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 19:12:14.888666 23148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.60:34679: Network error: Client connection negotiation failed: client connection to 127.0.203.60:34679: connect: Connection refused (error 111)
Jun 20 19:12:14 dist-test-slave-wd30 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for kudu/127.0.203.61@KRBTEST.COM
I20260620 19:12:14.894768 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.886809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:34143 (local address 127.0.0.1:47424)
0620 19:12:14.886877 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.886891 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:14.886943 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:14.887185 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:14.887187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:14.887355 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:14.887423 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:14.887425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.888376 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.888378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:14.889182 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:14.889187 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.889581 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.889583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:14.889664 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:14.890174 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:14.890179 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:14.891665 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:14.893563 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:14.893565 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:14.893565 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:14.893842 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:14.894063 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:14.894064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:14.894064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:14.894154 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:14.894410 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:14.894411 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:14.894466 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 19:12:14.894554 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 19:12:14.894667 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 19:12:14.895097 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.886937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.0.203.61:34143)
0620 19:12:14.887021 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:14.887023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:14.887032 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:12:14.887067 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:14.887068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:14.887077 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:14.887155 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:14.887541 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.888255 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.889277 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.889503 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.889583 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:14.891761 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:14.891763 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:14.891763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:14.891770 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:14.893469 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:14.893919 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:14.893919 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:14.893920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:14.893977 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:14.894228 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:14.894229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:14.894229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:14.894344 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:14.894403 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:14.894581 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 19:12:14.894648 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:12:14.895530 23073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:12:14.895897 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.895796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:34679 (local address 127.0.0.1:37434)
0620 19:12:14.895847 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.895878 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 19:12:14.896202 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.896116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:34679 (local address 127.0.0.1:37438)
0620 19:12:14.896158 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.896181 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 19:12:14.896287 23073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.60:34679: Network error: Client connection negotiation failed: client connection to 127.0.203.60:34679: connect: Connection refused (error 111)
W20260620 19:12:14.924041 23155 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:12:14.924331 23155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:14.924361 23155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:14.924408 23155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:14.925643 23155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:14.925724 23155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:14.925750 23155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:14.925783 23155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:14.925832 23155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:14.927425 23073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } attempt: 1
I20260620 19:12:14.927577 23155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:36485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:41907,127.0.203.61:34143,127.0.203.60:34679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/krb5kdc/kudu_127.0.203.60.keytab
--principal=kudu/127.0.203.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.60:34679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.203.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:14.927888 23155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:14.927915 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.927783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:34679 (local address 127.0.0.1:37442)
0620 19:12:14.927852 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.927885 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 19:12:14.928175 23155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:12:14.928272 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.928152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:34679 (local address 127.0.0.1:37446)
0620 19:12:14.928201 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.928232 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 19:12:14.928356 23073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.60:34679: Network error: Client connection negotiation failed: client connection to 127.0.203.60:34679: connect: Connection refused (error 111)
W20260620 19:12:14.933138 23160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:14.933104 23163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:14.933099 23161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:14.933375 23155 server_base.cc:1034] running on GCE node
I20260620 19:12:14.933650 23155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:14.933897 23155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:14.935039 23155 hybrid_clock.cc:648] HybridClock initialized: now 1781982734935017 us; error 35 us; skew 500 ppm
Jun 20 19:12:14 dist-test-slave-wd30 krb5kdc[22992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:14.937456 23155 init.cc:377] Logged in from keytab as kudu/127.0.203.60@KRBTEST.COM (short username kudu)
I20260620 19:12:14.937877 23155 webserver.cc:460] Webserver started at http://127.0.203.60:36877/ using document root <none> and password file <none>
I20260620 19:12:14.938112 23155 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:14.938177 23155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:14.938321 23155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:14.939435 23155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-2/data/instance:
uuid: "64dddc0cb2f44517b92ebffcf0fa576b"
format_stamp: "Formatted at 2026-06-20 19:12:14 on dist-test-slave-wd30"
server_key: "163737d3a017827eff133239ec87df08"
server_key_iv: "ea1d056fe2dcd95679ea27c1a829fc94"
server_key_version: "encryptionkey@0"
I20260620 19:12:14.939798 23155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-2/wal/instance:
uuid: "64dddc0cb2f44517b92ebffcf0fa576b"
format_stamp: "Formatted at 2026-06-20 19:12:14 on dist-test-slave-wd30"
server_key: "163737d3a017827eff133239ec87df08"
server_key_iv: "ea1d056fe2dcd95679ea27c1a829fc94"
server_key_version: "encryptionkey@0"
I20260620 19:12:14.941115 23155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:12:14.941835 23170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:14.942023 23155 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:12:14.942127 23155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-2/wal
uuid: "64dddc0cb2f44517b92ebffcf0fa576b"
format_stamp: "Formatted at 2026-06-20 19:12:14 on dist-test-slave-wd30"
server_key: "163737d3a017827eff133239ec87df08"
server_key_iv: "ea1d056fe2dcd95679ea27c1a829fc94"
server_key_version: "encryptionkey@0"
I20260620 19:12:14.942199 23155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:14.952411 23155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:14.954138 23155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:14.954321 23155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:14.956805 23148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } attempt: 2
I20260620 19:12:14.957226 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.957095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:34679 (local address 127.0.0.1:37458)
0620 19:12:14.957155 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.957191 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 19:12:14.957598 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.957465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:34679 (local address 127.0.0.1:37460)
0620 19:12:14.957524 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.957562 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 19:12:14.957711 23148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.60:34679: Network error: Client connection negotiation failed: client connection to 127.0.203.60:34679: connect: Connection refused (error 111)
I20260620 19:12:14.958559 23155 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.60:34679
I20260620 19:12:14.958585 23222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.60:34679 every 8 connection(s)
I20260620 19:12:14.958891 23155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-2/data/info.pb
I20260620 19:12:14.959858 23223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:14.961272 23223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:12:14.963070 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 23155
I20260620 19:12:14.963236 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982468536772-812-0/minicluster-data/master-2/wal/instance
I20260620 19:12:14.963487 812 external_mini_cluster.cc:1428] Setting key 3c1d1df98a3da854d5391813c6adf522
Jun 20 19:12:14 dist-test-slave-wd30 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.60@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
Jun 20 19:12:14 dist-test-slave-wd30 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.60@KRBTEST.COM
I20260620 19:12:14.971619 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.961869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.0.203.62:41907)
0620 19:12:14.961943 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:14.961946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:14.962115 (+ 169us) server_negotiation.cc:407] Connection header received
0620 19:12:14.962558 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:14.962559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:14.962569 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:14.962622 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:14.963558 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.964298 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.965536 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.965834 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.965912 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:14.968382 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:14.968384 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:14.968384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:14.968393 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:14.969996 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:14.970526 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:14.970527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:14.970527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:14.970611 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:14.970892 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:14.970893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:14.970894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:14.971045 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:14.971143 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:14.971439 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 19:12:14.971531 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:12:14.971820 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.961827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:41907 (local address 127.0.0.1:52760)
0620 19:12:14.962006 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.962038 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:14.962458 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:14.962731 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:14.962734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:14.963099 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:14.963442 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:14.963448 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.964392 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.964393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:14.965444 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:14.965447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.965952 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.965953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:14.966101 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:14.966630 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:14.966641 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:14.968284 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:14.970077 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:14.970084 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:14.970086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:14.970450 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:14.970695 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:14.970695 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:14.970696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:14.970821 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:14.971159 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:14.971166 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:14.971301 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 19:12:14.971393 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 19:12:14.971556 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":95,"threads_started":1}
I20260620 19:12:14.972434 23223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:12:14.972743 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.963881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37470 (local address 127.0.203.60:34679)
0620 19:12:14.964050 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:14.964054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:14.964105 (+ 51us) server_negotiation.cc:407] Connection header received
0620 19:12:14.964161 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:14.964163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:14.964182 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:14.964249 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:14.965009 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.965831 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.966811 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.967161 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.967231 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:14.969564 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:14.969572 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:14.969572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:14.969579 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:14.971439 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:14.971828 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:14.971832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:14.971832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:14.971892 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:14.972119 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:14.972120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:14.972121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:14.972287 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:14.972430 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:14.972526 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 19:12:14.972609 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":127,"threads_started":1}
Jun 20 19:12:14 dist-test-slave-wd30 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.60@KRBTEST.COM for kudu/127.0.203.61@KRBTEST.COM
Jun 20 19:12:14 dist-test-slave-wd30 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:12:14.980098 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.972733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:34143 (local address 127.0.0.1:47426)
0620 19:12:14.972887 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.972899 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:14.972952 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:14.973132 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:14.973133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:14.973276 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:14.973342 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:14.973343 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.974291 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.974291 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:14.974771 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:14.974772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.975225 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.975226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:14.975300 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:14.975742 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:14.975748 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:14.977170 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:14.979011 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:14.979012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:14.979013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:14.979275 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:14.979503 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:14.979504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:14.979504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:14.979573 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:14.979797 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:14.979798 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:14.979851 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 19:12:14.979936 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 19:12:14.980028 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 19:12:14.980188 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.972822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47426 (local address 127.0.203.61:34143)
0620 19:12:14.972870 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:14.972872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:14.973036 (+ 164us) server_negotiation.cc:407] Connection header received
0620 19:12:14.973061 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:14.973061 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:14.973071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:14.973112 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:14.973482 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.974205 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.974873 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.975099 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.975226 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:14.977265 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:14.977267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:14.977267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:14.977275 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:14.978904 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:14.979357 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:14.979359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:14.979359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:14.979412 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:14.979638 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:14.979638 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:14.979639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:14.979737 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:14.979783 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:14.979994 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 19:12:14.980072 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:12:14.980736 23223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:12:14.981254 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.973705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52774 (local address 127.0.203.62:41907)
0620 19:12:14.973751 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:14.973753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:14.973797 (+ 44us) server_negotiation.cc:407] Connection header received
0620 19:12:14.973865 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:14.973866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:14.973874 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:14.973911 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:14.974501 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.974948 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.975708 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.975910 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.976000 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:14.978245 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:14.978246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:14.978247 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:14.978254 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:14.979864 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:14.980260 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:14.980261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:14.980262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:14.980331 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:14.980505 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:14.980505 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:14.980505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:14.980635 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:14.980797 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:14.981046 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 19:12:14.981138 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 19:12:14 dist-test-slave-wd30 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.60@KRBTEST.COM for kudu/127.0.203.60@KRBTEST.COM
Jun 20 19:12:14 dist-test-slave-wd30 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.61@KRBTEST.COM
I20260620 19:12:14.988029 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.981087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.0.203.60:34679)
0620 19:12:14.981127 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:14.981129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:14.981136 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:12:14.981259 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:14.981260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:14.981273 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:14.981329 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:14.981628 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.982282 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.983037 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.983356 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.983407 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:14.985342 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:14.985344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:14.985345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:14.985353 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:14.986823 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:14.987228 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:14.987229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:14.987229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:14.987284 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:14.987491 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:14.987492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:14.987493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:14.987608 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:14.987679 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:14.987823 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 19:12:14.987905 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:12:14.988077 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.981034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:34679 (local address 127.0.0.1:37484)
0620 19:12:14.981081 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:14.981092 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:14.981126 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:14.981351 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:14.981352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:14.981523 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:14.981567 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:14.981569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.982360 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.982361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:14.982952 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:14.982954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.983424 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.983425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:14.983482 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:14.983859 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:14.983864 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:14.985257 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:14.986904 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:14.986905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:14.986905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:14.987157 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:14.987348 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:14.987349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:14.987349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:14.987406 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:14.987671 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:14.987671 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:14.987711 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 19:12:14.987799 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 19:12:14.987957 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":67}
I20260620 19:12:14.989801 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:14.981934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47432 (local address 127.0.203.61:34143)
0620 19:12:14.981987 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:14.981996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:14.982003 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:12:14.982023 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:14.982024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:14.982033 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:14.982091 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:14.982621 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.983098 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.983949 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:14.984172 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:14.984238 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:14.986839 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:14.986840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:14.986841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:14.986848 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:14.988597 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:14.988992 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:14.988993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:14.988993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:14.989058 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:14.989262 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:14.989263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:14.989264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:14.989388 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:14.989582 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:14.989621 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 19:12:14.989701 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 19:12:14.989874 23223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b: Bootstrap starting.
I20260620 19:12:14.991003 23223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:14.991394 23223 log.cc:824] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:14.991516 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:12:14.992218 23223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b: No bootstrap required, opened a new log
I20260620 19:12:14.992697 23047 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b47b27b19a484c5bac5f55faec750367"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52774
I20260620 19:12:14.994292 23223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47b27b19a484c5bac5f55faec750367" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } } peers { permanent_uuid: "a8cdf5225b614c3db6bae4db85edb728" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } } peers { permanent_uuid: "64dddc0cb2f44517b92ebffcf0fa576b" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } }
I20260620 19:12:14.994422 23223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64dddc0cb2f44517b92ebffcf0fa576b, State: Initialized, Role: FOLLOWER
I20260620 19:12:14.994524 23223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47b27b19a484c5bac5f55faec750367" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } } peers { permanent_uuid: "a8cdf5225b614c3db6bae4db85edb728" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } } peers { permanent_uuid: "64dddc0cb2f44517b92ebffcf0fa576b" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } }
I20260620 19:12:14.994872 23223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:14.994827 23229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47b27b19a484c5bac5f55faec750367" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } } peers { permanent_uuid: "a8cdf5225b614c3db6bae4db85edb728" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } } peers { permanent_uuid: "64dddc0cb2f44517b92ebffcf0fa576b" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } } }
I20260620 19:12:14.994951 23229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b [sys.catalog]: This master's current role is: FOLLOWER
W20260620 19:12:14.996798 23240 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:12:14.996856 23240 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:12:14.997253 23155 master_runner.cc:186] Error getting master registration for 127.0.203.62:41907: OK, instance_id { permanent_uuid: "b47b27b19a484c5bac5f55faec750367" instance_seqno: 1781982734656772 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:12:15.000432 23155 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:12:15.003487 23073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } attempt: 2
Jun 20 19:12:15 dist-test-slave-wd30 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for kudu/127.0.203.60@KRBTEST.COM
I20260620 19:12:15.011058 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:15.003930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37492 (local address 127.0.203.60:34679)
0620 19:12:15.003998 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:15.004001 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:15.004024 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:12:15.004049 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:15.004050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:15.004059 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:15.004106 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:15.004433 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:15.004957 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:15.005690 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:15.005906 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:15.005990 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:15.008179 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:15.008180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:15.008181 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:15.008189 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:15.009776 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:15.010169 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:15.010171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:15.010171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:15.010236 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:15.010445 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:15.010446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:15.010447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:15.010621 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:15.010704 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:15.010864 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 19:12:15.010956 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:12:15.011058 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:15.003844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:34679 (local address 127.0.0.1:37492)
0620 19:12:15.003908 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:15.003921 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:15.003977 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:15.004109 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:15.004110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:15.004310 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:15.004367 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:15.004369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:15.005055 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:15.005057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:15.005591 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:15.005593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:15.005989 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:15.005989 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:15.006053 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:15.006585 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:15.006591 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:15.008076 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:15.009885 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:15.009886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:15.009887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:15.010080 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:15.010306 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:15.010307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:15.010307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:15.010373 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:15.010695 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:15.010696 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:15.010748 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 19:12:15.010839 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 19:12:15.010952 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 19:12:15.012807 23073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367: Bootstrap starting.
I20260620 19:12:15.013696 23073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:15.014009 23073 log.cc:824] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:15.014827 23073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367: No bootstrap required, opened a new log
I20260620 19:12:15.016358 23073 raft_consensus.cc:348] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47b27b19a484c5bac5f55faec750367" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } } peers { permanent_uuid: "a8cdf5225b614c3db6bae4db85edb728" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } } peers { permanent_uuid: "64dddc0cb2f44517b92ebffcf0fa576b" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } }
I20260620 19:12:15.016522 23073 raft_consensus.cc:729] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b47b27b19a484c5bac5f55faec750367, State: Initialized, Role: FOLLOWER
I20260620 19:12:15.016599 23073 consensus_queue.cc:260] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47b27b19a484c5bac5f55faec750367" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } } peers { permanent_uuid: "a8cdf5225b614c3db6bae4db85edb728" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } } peers { permanent_uuid: "64dddc0cb2f44517b92ebffcf0fa576b" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } }
I20260620 19:12:15.016960 23073 sys_catalog.cc:564] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:15.016986 23047 raft_consensus.cc:482] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:12:15.016930 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47b27b19a484c5bac5f55faec750367" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } } peers { permanent_uuid: "a8cdf5225b614c3db6bae4db85edb728" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } } peers { permanent_uuid: "64dddc0cb2f44517b92ebffcf0fa576b" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } } }
I20260620 19:12:15.017107 23047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:15.017196 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:12:15.018154 23047 raft_consensus.cc:504] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47b27b19a484c5bac5f55faec750367" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } } peers { permanent_uuid: "a8cdf5225b614c3db6bae4db85edb728" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } } peers { permanent_uuid: "64dddc0cb2f44517b92ebffcf0fa576b" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } }
I20260620 19:12:15.019029 23197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b47b27b19a484c5bac5f55faec750367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "64dddc0cb2f44517b92ebffcf0fa576b"
I20260620 19:12:15.019279 23197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:15.020092 23197 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b47b27b19a484c5bac5f55faec750367 in term 1.
W20260620 19:12:15.020243 23254 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 19:12:15.020361 23024 leader_election.cc:302] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64dddc0cb2f44517b92ebffcf0fa576b, b47b27b19a484c5bac5f55faec750367; no voters:
I20260620 19:12:15.020512 23243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:15.020581 23047 leader_election.cc:288] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [CANDIDATE]: Term 1 election: Requested vote from peers a8cdf5225b614c3db6bae4db85edb728 (127.0.203.61:34143), 64dddc0cb2f44517b92ebffcf0fa576b (127.0.203.60:34679)
I20260620 19:12:15.020653 23243 raft_consensus.cc:686] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [term 1 LEADER]: Becoming Leader. State: Replica: b47b27b19a484c5bac5f55faec750367, State: Running, Role: LEADER
I20260620 19:12:15.020735 23243 consensus_queue.cc:237] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47b27b19a484c5bac5f55faec750367" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } } peers { permanent_uuid: "a8cdf5225b614c3db6bae4db85edb728" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } } peers { permanent_uuid: "64dddc0cb2f44517b92ebffcf0fa576b" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } }
I20260620 19:12:15.021128 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b47b27b19a484c5bac5f55faec750367. Latest consensus state: current_term: 1 leader_uuid: "b47b27b19a484c5bac5f55faec750367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47b27b19a484c5bac5f55faec750367" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } } peers { permanent_uuid: "a8cdf5225b614c3db6bae4db85edb728" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } } peers { permanent_uuid: "64dddc0cb2f44517b92ebffcf0fa576b" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } } }
I20260620 19:12:15.021193 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [sys.catalog]: This master's current role is: LEADER
W20260620 19:12:15.020391 23254 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:12:15.022143 23260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:15.022339 23260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:15.020774 23005 master_runner.cc:186] Error getting master registration for 127.0.203.61:34143: OK, instance_id { permanent_uuid: "a8cdf5225b614c3db6bae4db85edb728" instance_seqno: 1781982734808265 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:12:15.023540 23005 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:12:15.024672 23197 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b [term 1 FOLLOWER]: Refusing update from remote peer b47b27b19a484c5bac5f55faec750367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:15.025074 23256 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64dddc0cb2f44517b92ebffcf0fa576b" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:15.026659 23229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b47b27b19a484c5bac5f55faec750367. Latest consensus state: current_term: 1 leader_uuid: "b47b27b19a484c5bac5f55faec750367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47b27b19a484c5bac5f55faec750367" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } } peers { permanent_uuid: "a8cdf5225b614c3db6bae4db85edb728" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } } peers { permanent_uuid: "64dddc0cb2f44517b92ebffcf0fa576b" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } } }
I20260620 19:12:15.026748 23229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:12:15.033274 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:15.024984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37498 (local address 127.0.203.60:34679)
0620 19:12:15.025029 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:15.025031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:15.025436 (+ 405us) server_negotiation.cc:407] Connection header received
0620 19:12:15.025544 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:15.025545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:15.025555 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:15.025585 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:15.026400 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:15.027174 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:15.028213 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:15.028418 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:15.028520 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:15.030084 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:15.030085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:15.030085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:15.030092 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:15.031888 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:15.032394 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:15.032395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:15.032396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:15.032452 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:15.032684 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:15.032685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:15.032685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:15.032803 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:15.032891 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:15.033112 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 19:12:15.033185 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:12:15.037322 23256 sys_catalog.cc:455] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b47b27b19a484c5bac5f55faec750367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47b27b19a484c5bac5f55faec750367" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } } peers { permanent_uuid: "a8cdf5225b614c3db6bae4db85edb728" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } } peers { permanent_uuid: "64dddc0cb2f44517b92ebffcf0fa576b" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } } }
I20260620 19:12:15.037442 23256 sys_catalog.cc:458] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:15.038094 23229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b47b27b19a484c5bac5f55faec750367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47b27b19a484c5bac5f55faec750367" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } } peers { permanent_uuid: "a8cdf5225b614c3db6bae4db85edb728" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } } peers { permanent_uuid: "64dddc0cb2f44517b92ebffcf0fa576b" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } } }
I20260620 19:12:15.038183 23229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:12:15.038450 23263 mvcc.cc:205] Tried to move back new op lower bound from 7299001282658566144 to 7299001282645934080. Current Snapshot: MvccSnapshot[applied={T|T < 7299001282658566144}]
I20260620 19:12:15.039000 23260 catalog_manager.cc:1347] Generated new cluster ID: 3bf0e9d65f634e50a219fc09b84446aa
I20260620 19:12:15.039060 23260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:15.039469 23267 mvcc.cc:205] Tried to move back new op lower bound from 7299001282658566144 to 7299001282645934080. Current Snapshot: MvccSnapshot[applied={T|T < 7299001282658566144}]
I20260620 19:12:15.040078 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:15.024745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47448 (local address 127.0.203.61:34143)
0620 19:12:15.024794 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:15.024796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:15.031268 (+ 6472us) server_negotiation.cc:407] Connection header received
0620 19:12:15.031613 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:15.031615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:15.031626 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:15.031657 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:15.032693 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:15.033368 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:15.034622 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:15.034938 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:15.034998 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:15.036566 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:15.036567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:15.036567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:15.036574 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:15.038454 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:15.039078 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:15.039079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:15.039079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:15.039152 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:15.039438 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:15.039438 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:15.039439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:15.039561 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:15.039620 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:15.039912 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 19:12:15.039991 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:12:15.043414 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:15.024499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52778 (local address 127.0.203.62:41907)
0620 19:12:15.024547 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:15.024550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:15.031362 (+ 6812us) server_negotiation.cc:407] Connection header received
0620 19:12:15.031508 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:15.031509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:15.031520 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:15.031550 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:15.032858 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:15.033564 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:15.035041 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:15.035394 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:15.035448 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:15.037660 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:15.037661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:15.037662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:15.037668 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:15.039394 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:15.040598 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:15.040599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:15.040599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:15.040657 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:15.040894 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:15.040895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:15.040895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:15.041021 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:15.041105 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:15.043238 (+ 2133us) server_negotiation.cc:299] Negotiation successful
0620 19:12:15.043317 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:12:15.048877 23148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } attempt: 3
I20260620 19:12:15.053869 23260 catalog_manager.cc:1370] Generated new certificate authority record
Jun 20 19:12:15 dist-test-slave-wd30 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.61@KRBTEST.COM for kudu/127.0.203.60@KRBTEST.COM
I20260620 19:12:15.055030 23260 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:15.057986 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:15.049135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:34679 (local address 127.0.0.1:37508)
0620 19:12:15.049233 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:15.049243 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:15.049278 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:15.049501 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:15.049502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:15.049650 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:15.049690 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:15.049691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:15.051385 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:15.051385 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:15.052194 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:15.052196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:15.052684 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:15.052685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:15.052739 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:15.053248 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:15.053253 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:15.054778 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:15.056897 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:15.056899 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:15.056899 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:15.057122 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:15.057332 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:15.057333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:15.057333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:15.057398 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:15.057678 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:15.057679 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:15.057731 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 19:12:15.057789 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 19:12:15.057876 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 19:12:15.058652 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:15.049351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.0.203.60:34679)
0620 19:12:15.049389 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:15.049391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:15.049412 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:12:15.049431 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:15.049432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:15.049441 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:15.049498 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:15.049775 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:15.050530 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:15.052286 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:15.052585 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:15.052638 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:15.054885 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:15.054886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:15.054887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:15.054893 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:15.056803 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:15.057201 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:15.057202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:15.057202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:15.057263 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:15.057468 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:15.057469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:15.057469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:15.057594 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:15.057646 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:15.058399 (+ 753us) server_negotiation.cc:299] Negotiation successful
0620 19:12:15.058516 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:12:15.060613 23148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728: Bootstrap starting.
I20260620 19:12:15.061331 23148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:15.061651 23148 log.cc:824] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:15.062610 23148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728: No bootstrap required, opened a new log
I20260620 19:12:15.064730 23148 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47b27b19a484c5bac5f55faec750367" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } } peers { permanent_uuid: "a8cdf5225b614c3db6bae4db85edb728" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } } peers { permanent_uuid: "64dddc0cb2f44517b92ebffcf0fa576b" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } }
I20260620 19:12:15.064880 23148 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8cdf5225b614c3db6bae4db85edb728, State: Initialized, Role: FOLLOWER
I20260620 19:12:15.064971 23148 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47b27b19a484c5bac5f55faec750367" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } } peers { permanent_uuid: "a8cdf5225b614c3db6bae4db85edb728" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } } peers { permanent_uuid: "64dddc0cb2f44517b92ebffcf0fa576b" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } }
I20260620 19:12:15.065359 23148 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:15.065448 23121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:15.066510 23121 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728 [term 1 FOLLOWER]: Refusing update from remote peer b47b27b19a484c5bac5f55faec750367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:15.066910 23122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b47b27b19a484c5bac5f55faec750367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a8cdf5225b614c3db6bae4db85edb728"
I20260620 19:12:15.066977 23260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367: Generated new TSK 0
I20260620 19:12:15.067211 23260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:12:15.067840 23122 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b47b27b19a484c5bac5f55faec750367 in term 1.
I20260620 19:12:15.068192 23243 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8cdf5225b614c3db6bae4db85edb728" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:12:15.068842 23283 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:12:15.068940 23283 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:12:15.065305 23272 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47b27b19a484c5bac5f55faec750367" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } } peers { permanent_uuid: "a8cdf5225b614c3db6bae4db85edb728" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } } peers { permanent_uuid: "64dddc0cb2f44517b92ebffcf0fa576b" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } } }
I20260620 19:12:15.069137 23272 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:12:15.069717 23272 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b47b27b19a484c5bac5f55faec750367. Latest consensus state: current_term: 1 leader_uuid: "b47b27b19a484c5bac5f55faec750367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47b27b19a484c5bac5f55faec750367" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } } peers { permanent_uuid: "a8cdf5225b614c3db6bae4db85edb728" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } } peers { permanent_uuid: "64dddc0cb2f44517b92ebffcf0fa576b" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } } }
I20260620 19:12:15.069798 23272 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:12:15.069831 23272 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b47b27b19a484c5bac5f55faec750367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47b27b19a484c5bac5f55faec750367" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } } peers { permanent_uuid: "a8cdf5225b614c3db6bae4db85edb728" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } } peers { permanent_uuid: "64dddc0cb2f44517b92ebffcf0fa576b" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } } }
I20260620 19:12:15.069875 23272 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:12:15.070930 23286 mvcc.cc:205] Tried to move back new op lower bound from 7299001282802126848 to 7299001282645934080. Current Snapshot: MvccSnapshot[applied={T|T < 7299001282658566144}]
W20260620 19:12:15.094846 23227 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:15.093157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.0.203.60:34679)
0620 19:12:15.093244 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:15.093246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:15.094008 (+ 762us) server_negotiation.cc:407] Connection header received
0620 19:12:15.094142 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:15.094143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:15.094156 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:15.094189 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:15.094769 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37520: BlockingRecv error: recv got EOF from 127.0.0.1:37520 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 19:12:15.095736 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:15.093664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47454 (local address 127.0.203.61:34143)
0620 19:12:15.093707 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:15.093709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:15.095167 (+ 1458us) server_negotiation.cc:407] Connection header received
0620 19:12:15.095273 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:15.095274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:15.095285 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:15.095314 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:15.095638 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47454: BlockingRecv error: recv got EOF from 127.0.0.1:47454 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 19:12:15.096680 23077 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:15.096138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52790 (local address 127.0.203.62:41907)
0620 19:12:15.096187 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:15.096189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:15.096203 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:12:15.096251 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:15.096252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:15.096263 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:15.096344 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:15.096616 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52790: BlockingRecv error: recv got EOF from 127.0.0.1:52790 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260620 19:12:15.100577 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:15.099720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47458 (local address 127.0.203.61:34143)
0620 19:12:15.099761 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:15.099763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:15.099985 (+ 222us) server_negotiation.cc:407] Connection header received
0620 19:12:15.100104 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:15.100105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:15.100128 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:15.100159 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:15.100522 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47458: BlockingRecv error: recv got EOF from 127.0.0.1:47458 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 19:12:15.103840 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:15.102936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47470 (local address 127.0.203.61:34143)
0620 19:12:15.102988 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:15.102990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:15.103299 (+ 309us) server_negotiation.cc:407] Connection header received
0620 19:12:15.103400 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:15.103401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:15.103413 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:15.103442 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:15.103787 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47470: BlockingRecv error: recv got EOF from 127.0.0.1:47470 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 19:12:15.108207 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:15.107051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47480 (local address 127.0.203.61:34143)
0620 19:12:15.107091 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:15.107093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:15.107400 (+ 307us) server_negotiation.cc:407] Connection header received
0620 19:12:15.107507 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:15.107508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:15.107521 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:15.107551 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:15.107931 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47480: BlockingRecv error: recv got EOF from 127.0.0.1:47480 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 19:12:15.114559 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:15.113579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.0.203.61:34143)
0620 19:12:15.113636 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:15.113638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:15.113928 (+ 290us) server_negotiation.cc:407] Connection header received
0620 19:12:15.114043 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:15.114045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:15.114057 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:15.114091 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:15.114501 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47494: BlockingRecv error: recv got EOF from 127.0.0.1:47494 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 19:12:15.125064 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:15.124050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47500 (local address 127.0.203.61:34143)
0620 19:12:15.124097 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:15.124100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:15.124403 (+ 303us) server_negotiation.cc:407] Connection header received
0620 19:12:15.124534 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:15.124536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:15.124549 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:15.124581 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:15.124989 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47500: BlockingRecv error: recv got EOF from 127.0.0.1:47500 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 19:12:15.143422 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:15.142386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.0.203.61:34143)
0620 19:12:15.142440 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:15.142442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:15.142761 (+ 319us) server_negotiation.cc:407] Connection header received
0620 19:12:15.142870 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:15.142871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:15.142884 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:15.142914 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:15.143363 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47502: BlockingRecv error: recv got EOF from 127.0.0.1:47502 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 19:12:15.177477 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:15.176712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47514 (local address 127.0.203.61:34143)
0620 19:12:15.176760 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:15.176762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:15.176887 (+ 125us) server_negotiation.cc:407] Connection header received
0620 19:12:15.176988 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:15.176989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:15.177003 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:15.177079 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:15.177402 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47514: BlockingRecv error: recv got EOF from 127.0.0.1:47514 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 19:12:15.243309 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:15.242542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47524 (local address 127.0.203.61:34143)
0620 19:12:15.242590 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:15.242593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:15.242737 (+ 144us) server_negotiation.cc:407] Connection header received
0620 19:12:15.242809 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:15.242810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:15.242822 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:15.242891 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:15.243250 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47524: BlockingRecv error: recv got EOF from 127.0.0.1:47524 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 19:12:15.373330 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:15.372635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47536 (local address 127.0.203.61:34143)
0620 19:12:15.372704 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:15.372706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:15.372714 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:12:15.372741 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:15.372742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:15.372752 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:15.372817 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:15.373232 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47536: BlockingRecv error: recv got EOF from 127.0.0.1:47536 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:12:15.624971 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b47b27b19a484c5bac5f55faec750367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47b27b19a484c5bac5f55faec750367" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 41907 } } peers { permanent_uuid: "a8cdf5225b614c3db6bae4db85edb728" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 34143 } } peers { permanent_uuid: "64dddc0cb2f44517b92ebffcf0fa576b" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 34679 } } }
I20260620 19:12:15.625167 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P b47b27b19a484c5bac5f55faec750367 [sys.catalog]: This master's current role is: LEADER
W20260620 19:12:15.631208 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:15.630512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47552 (local address 127.0.203.61:34143)
0620 19:12:15.630588 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:15.630590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:15.630601 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:12:15.630660 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:15.630661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:15.630677 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:15.630748 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:15.631124 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47552: BlockingRecv error: recv got EOF from 127.0.0.1:47552 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:12:15.997336 23240 catalog_manager.cc:1259] Loaded cluster ID: 3bf0e9d65f634e50a219fc09b84446aa
I20260620 19:12:15.997452 23240 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b: loading cluster ID for follower catalog manager: success
I20260620 19:12:15.998276 23240 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b: acquiring CA information for follower catalog manager: success
I20260620 19:12:15.998620 23240 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 64dddc0cb2f44517b92ebffcf0fa576b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:12:16.069479 23283 catalog_manager.cc:1259] Loaded cluster ID: 3bf0e9d65f634e50a219fc09b84446aa
I20260620 19:12:16.069577 23283 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728: loading cluster ID for follower catalog manager: success
I20260620 19:12:16.070438 23283 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728: acquiring CA information for follower catalog manager: success
I20260620 19:12:16.070773 23283 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a8cdf5225b614c3db6bae4db85edb728: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:12:16.147339 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:16.144395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47562 (local address 127.0.203.61:34143)
0620 19:12:16.144555 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:16.144558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:16.144591 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:12:16.144666 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:16.144667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:16.144754 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:16.144844 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:16.144972 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:16.145485 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:16.146467 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:16.146712 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:16.146765 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:16.146978 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:16.147083 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:16.147159 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:16.147218 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 19:12:16.147234 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":64,"threads_started":1}
I20260620 19:12:16.151587 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:16.149151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37528 (local address 127.0.203.60:34679)
0620 19:12:16.149248 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:16.149250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:16.149259 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:12:16.149295 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:16.149295 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:16.149328 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:16.149416 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:16.149483 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:16.149939 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:16.150859 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:16.151055 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:16.151108 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:16.151326 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:16.151422 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:16.151475 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:16.151513 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 19:12:16.151529 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70,"thread_start_us":48,"threads_started":1}
I20260620 19:12:16.156136 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:16.153284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47564 (local address 127.0.203.61:34143)
0620 19:12:16.153326 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:16.153327 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:16.153544 (+ 217us) server_negotiation.cc:407] Connection header received
0620 19:12:16.153650 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:16.153650 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:16.153662 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:16.153692 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:16.153871 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:16.154315 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:16.155357 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:16.155629 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:16.155686 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:16.155837 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:16.155869 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:16.155919 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:16.156023 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 19:12:16.156046 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:12:16.156373 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:16.153507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37538 (local address 127.0.203.60:34679)
0620 19:12:16.153540 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:16.153542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:16.153600 (+ 58us) server_negotiation.cc:407] Connection header received
0620 19:12:16.153690 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:16.153691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:16.153700 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:16.153752 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:16.153898 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:16.154580 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:16.155583 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:16.155861 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:16.155955 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:16.156087 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:16.156117 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:16.156166 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:16.156294 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 19:12:16.156314 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:12:16.156589 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:16.153336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.0.203.62:41907)
0620 19:12:16.153463 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:16.153465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:16.153473 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:12:16.153522 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:16.153523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:16.153573 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:16.153660 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:16.153781 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:16.154559 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:16.155420 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:16.155714 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:16.155809 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:16.156123 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:16.156220 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:16.156464 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:16.156485 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 19:12:16.156509 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260620 19:12:16.158416 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 23005
I20260620 19:12:16.163745 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 23080
I20260620 19:12:16.168839 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 23155
2026-06-20T19:12:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2785 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:16 dist-test-slave-wd30 krb5kdc[23374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:16 dist-test-slave-wd30 krb5kdc[23374](info): set up 2 sockets
Jun 20 19:12:16 dist-test-slave-wd30 krb5kdc[23374](info): commencing operation
krb5kdc: starting...
W20260620 19:12:17.231679 812 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/test-user.keytab.
Jun 20 19:12:17 dist-test-slave-wd30 krb5kdc[23374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
Entry for principal kudu/127.0.203.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab.
I20260620 19:12:17.349900 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:39789
--webserver_interface=127.0.203.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:32941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:39789,127.0.203.61:41379,127.0.203.60:37447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:17.432269 23387 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:12:17.432530 23387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:17.432571 23387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:17.432610 23387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:17.433789 23387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:17.433861 23387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:17.433887 23387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:17.433915 23387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:17.433939 23387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:17.435506 23387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:32941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:39789,127.0.203.61:41379,127.0.203.60:37447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/kudu_127.0.203.62.keytab
--principal=kudu/127.0.203.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.62:39789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:17.435775 23387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:17.436010 23387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:17.440714 23392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:17.440729 23393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:17.440716 23395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:17.440912 23387 server_base.cc:1034] running on GCE node
I20260620 19:12:17.441161 23387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:17.441461 23387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:17.442624 23387 hybrid_clock.cc:648] HybridClock initialized: now 1781982737442610 us; error 47 us; skew 500 ppm
Jun 20 19:12:17 dist-test-slave-wd30 krb5kdc[23374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:17.448503 23387 init.cc:377] Logged in from keytab as kudu/127.0.203.62@KRBTEST.COM (short username kudu)
I20260620 19:12:17.448972 23387 webserver.cc:460] Webserver started at http://127.0.203.62:42973/ using document root <none> and password file <none>
I20260620 19:12:17.449183 23387 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:17.449223 23387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:17.449307 23387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:17.450299 23387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-0/data/instance:
uuid: "fa81118459ea4cb68d970825cb731ad9"
format_stamp: "Formatted at 2026-06-20 19:12:17 on dist-test-slave-wd30"
server_key: "08fa6a95ff618a7f1291a12ae9db4042"
server_key_iv: "30599f4c2367fa348fabe70ed28e7b9d"
server_key_version: "encryptionkey@0"
I20260620 19:12:17.450598 23387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-0/wal/instance:
uuid: "fa81118459ea4cb68d970825cb731ad9"
format_stamp: "Formatted at 2026-06-20 19:12:17 on dist-test-slave-wd30"
server_key: "08fa6a95ff618a7f1291a12ae9db4042"
server_key_iv: "30599f4c2367fa348fabe70ed28e7b9d"
server_key_version: "encryptionkey@0"
I20260620 19:12:17.451936 23387 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:12:17.452642 23402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:17.452800 23387 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:17.452901 23387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-0/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-0/wal
uuid: "fa81118459ea4cb68d970825cb731ad9"
format_stamp: "Formatted at 2026-06-20 19:12:17 on dist-test-slave-wd30"
server_key: "08fa6a95ff618a7f1291a12ae9db4042"
server_key_iv: "30599f4c2367fa348fabe70ed28e7b9d"
server_key_version: "encryptionkey@0"
I20260620 19:12:17.452953 23387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:17.467847 23387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:17.470352 23387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:17.470475 23387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:17.474653 23387 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:39789
I20260620 19:12:17.474712 23454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:39789 every 8 connection(s)
I20260620 19:12:17.475139 23387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:17.476188 23455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:17.477833 23455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:12:17 dist-test-slave-wd30 krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
I20260620 19:12:17.484652 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 23387
I20260620 19:12:17.484771 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-0/wal/instance
I20260620 19:12:17.485002 812 external_mini_cluster.cc:1428] Setting key 22d040bfd54ba05538bb8b00c3f16a68
I20260620 19:12:17.488070 23458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.478627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.0.203.62:39789)
0620 19:12:17.478922 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.478931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.478998 (+ 67us) server_negotiation.cc:407] Connection header received
0620 19:12:17.479603 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.479606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.479624 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.479702 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.480228 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.481042 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.481963 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.482317 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.482404 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.484647 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:17.484656 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:17.484656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:17.484665 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:17.486366 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.486865 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.486868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.486868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.486956 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.487244 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.487244 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.487245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.487466 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:17.487533 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:17.487764 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 19:12:17.487901 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":264,"thread_start_us":221,"threads_started":1}
I20260620 19:12:17.488188 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.478594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:39789 (local address 127.0.0.1:56730)
0620 19:12:17.478882 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.478929 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:17.479457 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:17.479720 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:17.479723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:17.479984 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:17.480143 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:17.480147 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.481153 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.481154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:17.481866 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:17.481869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.482414 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.482415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:17.482468 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.483167 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:17.483176 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:17.484548 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:17.486452 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:17.486460 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:17.486463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:17.486766 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:17.487030 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:17.487031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:17.487032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:17.487174 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:17.487542 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:17.487549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:17.487630 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 19:12:17.487734 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 19:12:17.487998 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":78,"thread_start_us":80,"threads_started":1}
I20260620 19:12:17.488943 23455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:12:17.489336 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.489198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:41379 (local address 127.0.0.1:57200)
0620 19:12:17.489251 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.489299 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.61:41379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 19:12:17.489450 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.203.61:41379: connect: Connection refused (error 111)
I20260620 19:12:17.489887 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.489787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:41379 (local address 127.0.0.1:57216)
0620 19:12:17.489835 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.489859 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.61:41379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 19:12:17.490017 23455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.61:41379: Network error: Client connection negotiation failed: client connection to 127.0.203.61:41379: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.203.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/kudu_127.0.203.61.keytab.
Entry for principal kudu/127.0.203.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/kudu_127.0.203.61.keytab.
I20260620 19:12:17.510838 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.61:41379
--webserver_interface=127.0.203.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/kudu_127.0.203.61.keytab
--principal=kudu/127.0.203.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:32941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:39789,127.0.203.61:41379,127.0.203.60:37447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:12:17.539276 23455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } attempt: 1
I20260620 19:12:17.539739 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.539610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:41379 (local address 127.0.0.1:57226)
0620 19:12:17.539671 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.539709 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.61:41379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 19:12:17.540060 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.539969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:41379 (local address 127.0.0.1:57240)
0620 19:12:17.540013 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.540036 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.61:41379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 19:12:17.540139 23455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.61:41379: Network error: Client connection negotiation failed: client connection to 127.0.203.61:41379: connect: Connection refused (error 111)
W20260620 19:12:17.592566 23462 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:12:17.592859 23462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:17.592893 23462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:17.592937 23462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:17.594187 23462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:17.594275 23462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:17.594303 23462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:17.594336 23462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:17.594358 23462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:17.596160 23462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:32941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:39789,127.0.203.61:41379,127.0.203.60:37447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/kudu_127.0.203.61.keytab
--principal=kudu/127.0.203.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.61:41379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.203.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:17.596467 23462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:17.596721 23462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:17.601776 23468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:17.601766 23470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:17.601766 23467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:17.601958 23462 server_base.cc:1034] running on GCE node
I20260620 19:12:17.602226 23462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:17.602550 23462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:17.603719 23462 hybrid_clock.cc:648] HybridClock initialized: now 1781982737603706 us; error 31 us; skew 500 ppm
Jun 20 19:12:17 dist-test-slave-wd30 krb5kdc[23374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:17.606362 23462 init.cc:377] Logged in from keytab as kudu/127.0.203.61@KRBTEST.COM (short username kudu)
I20260620 19:12:17.606812 23462 webserver.cc:460] Webserver started at http://127.0.203.61:40625/ using document root <none> and password file <none>
I20260620 19:12:17.607056 23462 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:17.607103 23462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:17.607277 23462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:17.608289 23455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } attempt: 2
I20260620 19:12:17.608454 23462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-1/data/instance:
uuid: "0d894703052144218e5b46b46e76b16e"
format_stamp: "Formatted at 2026-06-20 19:12:17 on dist-test-slave-wd30"
server_key: "37d3203793698e9ae801e19c49b77f55"
server_key_iv: "4d0cad3351e2ce3ee72249f569c1805f"
server_key_version: "encryptionkey@0"
I20260620 19:12:17.608734 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.608588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:41379 (local address 127.0.0.1:57242)
0620 19:12:17.608655 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.608693 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.61:41379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 19:12:17.608867 23462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-1/wal/instance:
uuid: "0d894703052144218e5b46b46e76b16e"
format_stamp: "Formatted at 2026-06-20 19:12:17 on dist-test-slave-wd30"
server_key: "37d3203793698e9ae801e19c49b77f55"
server_key_iv: "4d0cad3351e2ce3ee72249f569c1805f"
server_key_version: "encryptionkey@0"
I20260620 19:12:17.609115 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.609030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:41379 (local address 127.0.0.1:57256)
0620 19:12:17.609063 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.609091 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.61:41379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260620 19:12:17.609239 23455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.61:41379: Network error: Client connection negotiation failed: client connection to 127.0.203.61:41379: connect: Connection refused (error 111)
I20260620 19:12:17.610347 23462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:12:17.611049 23477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:17.611244 23462 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:17.611362 23462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-1/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-1/wal
uuid: "0d894703052144218e5b46b46e76b16e"
format_stamp: "Formatted at 2026-06-20 19:12:17 on dist-test-slave-wd30"
server_key: "37d3203793698e9ae801e19c49b77f55"
server_key_iv: "4d0cad3351e2ce3ee72249f569c1805f"
server_key_version: "encryptionkey@0"
I20260620 19:12:17.611441 23462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:17.627935 23462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:17.629937 23462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:17.630084 23462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:17.634083 23462 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.61:41379
I20260620 19:12:17.634155 23529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.61:41379 every 8 connection(s)
I20260620 19:12:17.634500 23462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-1/data/info.pb
I20260620 19:12:17.635515 23530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:17.636188 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 23462
I20260620 19:12:17.636302 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-1/wal/instance
I20260620 19:12:17.636488 812 external_mini_cluster.cc:1428] Setting key 1df90a1db943a4b0c22bcbb6639d557f
I20260620 19:12:17.637823 23530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.203.60@KRBTEST.COM; defaulting to no policy
Jun 20 19:12:17 dist-test-slave-wd30 krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.61@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.60@KRBTEST.COM" created.
I20260620 19:12:17.646979 23458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.638702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56740 (local address 127.0.203.62:39789)
0620 19:12:17.638762 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.638764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.638943 (+ 179us) server_negotiation.cc:407] Connection header received
0620 19:12:17.639511 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.639512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.639527 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.639592 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.640267 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.640758 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.641743 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.641963 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.642041 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.644322 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:17.644329 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:17.644329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:17.644336 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:17.645484 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.645975 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.645976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.645976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.646036 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.646343 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.646344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.646345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.646455 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:17.646514 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:17.646851 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 19:12:17.646915 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:12:17.647392 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.638645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:39789 (local address 127.0.0.1:56740)
0620 19:12:17.638816 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.638859 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:17.639405 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:17.639677 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:17.639680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:17.639952 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:17.640178 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:17.640183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.640856 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.640857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:17.641639 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:17.641641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.642058 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.642059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:17.642184 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.642679 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:17.642687 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:17.644162 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:17.645552 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:17.645558 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:17.645559 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:17.645867 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:17.646102 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:17.646103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:17.646103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:17.646237 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:17.646528 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:17.646535 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:17.646711 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 19:12:17.646827 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 19:12:17.647005 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260620 19:12:17.647908 23530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:12:17 dist-test-slave-wd30 krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.61@KRBTEST.COM for kudu/127.0.203.61@KRBTEST.COM
I20260620 19:12:17.655170 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.648201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:41379 (local address 127.0.0.1:57260)
0620 19:12:17.648253 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.648263 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:17.648311 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:17.648540 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:17.648541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:17.648675 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:17.648716 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:17.648717 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.649656 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.649657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:17.650156 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:17.650157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.650632 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.650633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:17.650696 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.651136 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:17.651142 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:17.652278 (+ 1136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:17.653973 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:17.653974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:17.653974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:17.654176 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:17.654398 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:17.654399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:17.654399 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:17.654453 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:17.654780 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:17.654782 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:17.654839 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 19:12:17.654901 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:12:17.655009 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 19:12:17.655195 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.648244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57260 (local address 127.0.203.61:41379)
0620 19:12:17.648399 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.648403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.648422 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:12:17.648448 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.648449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.648459 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.648540 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.648804 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.649557 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.650235 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.650548 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.650626 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.652392 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:17.652399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:17.652399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:17.652406 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:17.653879 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.654272 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.654274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.654275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.654331 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.654563 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.654563 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.654564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.654704 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:17.654798 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:17.654945 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 19:12:17.655079 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260620 19:12:17.655987 23530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:12:17.656396 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.656264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:37447 (local address 127.0.0.1:41344)
0620 19:12:17.656311 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.656351 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260620 19:12:17.656550 23480 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.203.60:37447: connect: Connection refused (error 111)
I20260620 19:12:17.657019 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.656864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:37447 (local address 127.0.0.1:41356)
0620 19:12:17.656914 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.656969 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 19:12:17.657145 23530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.60:37447: Network error: Client connection negotiation failed: client connection to 127.0.203.60:37447: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/kudu_127.0.203.60.keytab.
Entry for principal kudu/127.0.203.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/kudu_127.0.203.60.keytab.
I20260620 19:12:17.660013 812 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeuOXoM/build/release/bin/kudu
/tmp/dist-test-taskeuOXoM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.60:37447
--webserver_interface=127.0.203.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/kudu_127.0.203.60.keytab
--principal=kudu/127.0.203.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.20:32941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.62:39789,127.0.203.61:41379,127.0.203.60:37447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:12:17.700421 23455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } attempt: 3
Jun 20 19:12:17 dist-test-slave-wd30 krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for kudu/127.0.203.61@KRBTEST.COM
I20260620 19:12:17.706385 23530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } attempt: 1
I20260620 19:12:17.706862 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.706715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:37447 (local address 127.0.0.1:41366)
0620 19:12:17.706783 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.706823 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 19:12:17.707329 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.707181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:37447 (local address 127.0.0.1:41378)
0620 19:12:17.707280 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.707305 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 19:12:17.707525 23530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.60:37447: Network error: Client connection negotiation failed: client connection to 127.0.203.60:37447: connect: Connection refused (error 111)
I20260620 19:12:17.708313 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.700941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57274 (local address 127.0.203.61:41379)
0620 19:12:17.701021 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.701024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.701037 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:12:17.701068 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.701070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.701083 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.701159 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.701548 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.702052 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.702794 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.703038 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.703122 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.705378 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:17.705380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:17.705380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:17.705396 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:17.707029 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.707457 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.707458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.707458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.707521 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.707734 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.707735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.707735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.707883 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:17.707964 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:17.708130 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 19:12:17.708220 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:12:17.708343 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.700860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:41379 (local address 127.0.0.1:57274)
0620 19:12:17.700914 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.700928 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:17.701009 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:17.701166 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:17.701167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:17.701348 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:17.701430 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:17.701433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.702152 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.702154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:17.702688 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:17.702690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.703153 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.703154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:17.703242 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.703780 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:17.703799 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:17.705268 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:17.707154 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:17.707156 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:17.707157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:17.707392 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:17.707585 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:17.707587 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:17.707587 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:17.707662 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:17.707957 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:17.707958 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:17.708011 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 19:12:17.708105 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 19:12:17.708220 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 19:12:17.709859 23455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:12:17.710224 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.710111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:37447 (local address 127.0.0.1:41388)
0620 19:12:17.710165 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.710198 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 19:12:17.710526 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.710441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:37447 (local address 127.0.0.1:41400)
0620 19:12:17.710485 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.710509 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 19:12:17.710655 23455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.60:37447: Network error: Client connection negotiation failed: client connection to 127.0.203.60:37447: connect: Connection refused (error 111)
I20260620 19:12:17.741830 23455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } attempt: 1
I20260620 19:12:17.742372 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.742196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:37447 (local address 127.0.0.1:41410)
0620 19:12:17.742266 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.742312 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 19:12:17.742771 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.742668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:37447 (local address 127.0.0.1:41416)
0620 19:12:17.742714 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.742745 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 19:12:17.742916 23455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.60:37447: Network error: Client connection negotiation failed: client connection to 127.0.203.60:37447: connect: Connection refused (error 111)
W20260620 19:12:17.745015 23537 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:12:17.745236 23537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:17.745271 23537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:17.745311 23537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:17.746439 23537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:17.746500 23537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:17.746526 23537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:17.746553 23537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:17.746580 23537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:17.748106 23537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.20:32941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.62:39789,127.0.203.61:41379,127.0.203.60:37447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/krb5kdc/kudu_127.0.203.60.keytab
--principal=kudu/127.0.203.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.60:37447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.203.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:17.748378 23537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:17.748617 23537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:17.753326 23545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:17.753324 23543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:17.753351 23542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:17.753350 23537 server_base.cc:1034] running on GCE node
I20260620 19:12:17.753911 23537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:17.754191 23537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:17.755350 23537 hybrid_clock.cc:648] HybridClock initialized: now 1781982737755344 us; error 33 us; skew 500 ppm
Jun 20 19:12:17 dist-test-slave-wd30 krb5kdc[23374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:17.757786 23537 init.cc:377] Logged in from keytab as kudu/127.0.203.60@KRBTEST.COM (short username kudu)
I20260620 19:12:17.758231 23537 webserver.cc:460] Webserver started at http://127.0.203.60:46253/ using document root <none> and password file <none>
I20260620 19:12:17.758409 23537 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:17.758448 23537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:17.758534 23537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:17.759575 23537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-2/data/instance:
uuid: "c446a54e3a9d40d381c4f46845949d93"
format_stamp: "Formatted at 2026-06-20 19:12:17 on dist-test-slave-wd30"
server_key: "530191277dba49c298224c7130c89fe7"
server_key_iv: "55a123af08725a0cf1b8907b593088f1"
server_key_version: "encryptionkey@0"
I20260620 19:12:17.759879 23537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-2/wal/instance:
uuid: "c446a54e3a9d40d381c4f46845949d93"
format_stamp: "Formatted at 2026-06-20 19:12:17 on dist-test-slave-wd30"
server_key: "530191277dba49c298224c7130c89fe7"
server_key_iv: "55a123af08725a0cf1b8907b593088f1"
server_key_version: "encryptionkey@0"
I20260620 19:12:17.761137 23537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:12:17.761807 23552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:17.762081 23537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:17.762152 23537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-2/data,/tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-2/wal
uuid: "c446a54e3a9d40d381c4f46845949d93"
format_stamp: "Formatted at 2026-06-20 19:12:17 on dist-test-slave-wd30"
server_key: "530191277dba49c298224c7130c89fe7"
server_key_iv: "55a123af08725a0cf1b8907b593088f1"
server_key_version: "encryptionkey@0"
I20260620 19:12:17.762233 23537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:17.775691 23530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } attempt: 2
I20260620 19:12:17.776196 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.776052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:37447 (local address 127.0.0.1:41430)
0620 19:12:17.776118 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.776162 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 19:12:17.776638 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.776544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:37447 (local address 127.0.0.1:41446)
0620 19:12:17.776592 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.776619 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 19:12:17.776768 23530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.60:37447: Network error: Client connection negotiation failed: client connection to 127.0.203.60:37447: connect: Connection refused (error 111)
I20260620 19:12:17.794968 23537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:17.797600 23537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:17.797758 23537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:17.802074 23537 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.60:37447
I20260620 19:12:17.802110 23604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.60:37447 every 8 connection(s)
I20260620 19:12:17.802513 23537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-2/data/info.pb
I20260620 19:12:17.803555 23605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:17.805148 812 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeuOXoM/build/release/bin/kudu as pid 23537
I20260620 19:12:17.805265 812 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeuOXoM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982468536772-812-0/minicluster-data/master-2/wal/instance
I20260620 19:12:17.805483 812 external_mini_cluster.cc:1428] Setting key 792bbb0d579063e8b208665b1ae2b5cd
I20260620 19:12:17.805464 23605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:12:17 dist-test-slave-wd30 krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.60@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
Jun 20 19:12:17 dist-test-slave-wd30 krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.60@KRBTEST.COM
I20260620 19:12:17.815202 23458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.806408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56752 (local address 127.0.203.62:39789)
0620 19:12:17.806462 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.806465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.806602 (+ 137us) server_negotiation.cc:407] Connection header received
0620 19:12:17.806882 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.806884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.806896 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.806944 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.807653 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.808143 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.809060 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.809267 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.809358 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.811763 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:17.811765 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:17.811765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:17.811773 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:17.813607 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.814036 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.814038 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.814038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.814101 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.814471 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.814473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.814473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.814611 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:17.814689 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:17.814987 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 19:12:17.815078 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 19:12:17.815430 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.806335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.62:39789 (local address 127.0.0.1:56752)
0620 19:12:17.806483 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.806529 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:17.806780 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:17.806963 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:17.806965 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:17.807379 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:17.807552 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:17.807557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.808231 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.808232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:17.808964 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:17.808966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.809388 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.809390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:17.809500 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.810047 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:17.810058 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:17.811665 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:17.813706 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:17.813714 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:17.813717 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:17.813958 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:17.814249 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:17.814250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:17.814250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:17.814384 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:17.814693 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:17.814699 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:17.814838 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 19:12:17.814963 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 19:12:17.815147 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260620 19:12:17.815788 23608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.806039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.0.203.60:37447)
0620 19:12:17.806216 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.806226 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.806259 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:12:17.806766 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.806773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.806795 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.806913 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.807658 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.808470 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.809234 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.809548 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.809630 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.812358 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:17.812367 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:17.812368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:17.812375 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:17.814309 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.814659 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.814663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.814663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.814724 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.814961 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.814962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.814962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.815168 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:17.815577 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:17.815631 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 19:12:17.815708 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260620 19:12:17.816160 23605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:12:17.818071 23455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } attempt: 2
Jun 20 19:12:17 dist-test-slave-wd30 krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.60@KRBTEST.COM for kudu/127.0.203.61@KRBTEST.COM
Jun 20 19:12:17 dist-test-slave-wd30 krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.62@KRBTEST.COM
Jun 20 19:12:17 dist-test-slave-wd30 krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.62@KRBTEST.COM for kudu/127.0.203.60@KRBTEST.COM
I20260620 19:12:17.825207 23458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.816813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.0.203.62:39789)
0620 19:12:17.816849 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.816851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.816856 (+ 5us) server_negotiation.cc:407] Connection header received
0620 19:12:17.816872 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.816872 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.816879 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.816946 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.817415 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.817879 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.818803 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.819051 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.819133 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.822252 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:17.822254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:17.822254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:17.822261 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:17.823952 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.824363 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.824364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.824365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.824425 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.824636 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.824637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.824638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.824758 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:17.824971 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:17.825008 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 19:12:17.825082 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:12:17.825377 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.816513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57286 (local address 127.0.203.61:41379)
0620 19:12:17.816627 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.816629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.816636 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:12:17.816660 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.816661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.816671 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.816743 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.817027 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.817499 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.818241 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.818584 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.818628 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.821564 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:17.821566 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:17.821566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:17.821574 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:17.823250 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.824442 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.824443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.824444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.824505 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.824705 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.824706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.824707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.824896 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:17.824944 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:17.825160 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 19:12:17.825247 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 19:12:17.825672 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.818310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:37447 (local address 127.0.0.1:41462)
0620 19:12:17.818353 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.818362 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:17.818392 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:17.818534 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:17.818535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:17.818688 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:17.818725 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:17.818726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.819630 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.819631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:17.820354 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:17.820357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.820832 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.820833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:17.820879 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.821288 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:17.821293 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:17.822662 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:17.824604 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:17.824604 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:17.824605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:17.824811 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:17.825032 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:17.825032 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:17.825033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:17.825099 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:17.825367 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:17.825367 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:17.825409 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 19:12:17.825460 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:12:17.825565 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 19:12:17.826016 23608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.818359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41462 (local address 127.0.203.60:37447)
0620 19:12:17.818393 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.818395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.818403 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:12:17.818452 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.818453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.818462 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.818511 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.818810 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.819561 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.820450 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.820761 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.820829 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.822757 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:17.822758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:17.822759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:17.822766 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:17.824509 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.824903 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.824905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.824905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.824958 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.825174 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.825175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.825175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.825290 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:17.825775 (+ 485us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:17.825831 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 19:12:17.825925 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:12:17.826262 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.816449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.61:41379 (local address 127.0.0.1:57286)
0620 19:12:17.816512 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.816523 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:17.816562 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:17.816746 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:17.816747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:17.816911 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:17.816947 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:17.816948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.817579 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.817580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:17.818138 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:17.818141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.818688 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.818689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:17.818734 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.819387 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:17.819391 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:17.821477 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:17.823384 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:17.824041 (+ 657us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:17.824042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:17.824335 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:17.824560 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:17.824561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:17.824562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:17.824620 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:17.824993 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:17.824994 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:17.825040 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 19:12:17.825943 (+ 903us) client_negotiation.cc:240] Negotiation successful
0620 19:12:17.826029 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 19:12:17.826918 23605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:12:17.827847 23455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9: Bootstrap starting.
I20260620 19:12:17.828737 23455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:17.829069 23455 log.cc:824] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:17.830052 23455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9: No bootstrap required, opened a new log
Jun 20 19:12:17 dist-test-slave-wd30 krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.61@KRBTEST.COM
Jun 20 19:12:17 dist-test-slave-wd30 krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.60@KRBTEST.COM for kudu/127.0.203.60@KRBTEST.COM
I20260620 19:12:17.832573 23455 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa81118459ea4cb68d970825cb731ad9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } } peers { permanent_uuid: "0d894703052144218e5b46b46e76b16e" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } } peers { permanent_uuid: "c446a54e3a9d40d381c4f46845949d93" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } }
I20260620 19:12:17.832738 23455 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa81118459ea4cb68d970825cb731ad9, State: Initialized, Role: FOLLOWER
I20260620 19:12:17.832836 23455 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa81118459ea4cb68d970825cb731ad9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } } peers { permanent_uuid: "0d894703052144218e5b46b46e76b16e" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } } peers { permanent_uuid: "c446a54e3a9d40d381c4f46845949d93" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } }
I20260620 19:12:17.833221 23455 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:17.833289 23611 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa81118459ea4cb68d970825cb731ad9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } } peers { permanent_uuid: "0d894703052144218e5b46b46e76b16e" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } } peers { permanent_uuid: "c446a54e3a9d40d381c4f46845949d93" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } } }
I20260620 19:12:17.833369 23611 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:12:17.834980 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.825853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57296 (local address 127.0.203.61:41379)
0620 19:12:17.825891 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.825893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.825951 (+ 58us) server_negotiation.cc:407] Connection header received
0620 19:12:17.826041 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.826042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.826052 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.826102 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.826737 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.827476 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.828133 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.828364 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.828455 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.831394 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:17.831396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:17.831396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:17.831404 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:17.833116 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.834055 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.834057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.834058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.834129 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.834378 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.834379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.834380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.834542 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:17.834752 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:17.834798 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 19:12:17.834876 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 19:12:17.835460 23622 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:12:17.835529 23622 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:12:17.836378 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.827219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:37447 (local address 127.0.0.1:41466)
0620 19:12:17.827270 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.827279 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:17.827313 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:17.829476 (+ 2163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:17.829477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:17.829624 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:17.829652 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:17.829654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.830312 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.830313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:17.830794 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:17.830795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.831208 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.831208 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:17.831257 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.831659 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:17.831664 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:17.832955 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:17.834706 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:17.834707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:17.834707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:17.835451 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:17.835741 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:17.835741 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:17.835742 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:17.835795 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:17.836101 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:17.836102 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:17.836136 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 19:12:17.836191 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 19:12:17.836286 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 19:12:17.836385 812 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:12:17.836906 23608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.827241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41466 (local address 127.0.203.60:37447)
0620 19:12:17.829337 (+ 2096us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.829339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.829350 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:12:17.829369 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.829370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.829380 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.829438 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.829729 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.830228 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.830903 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.831089 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.831204 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.833037 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:17.833038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:17.833038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:17.833044 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:17.834633 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.835615 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.835616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.835617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.835673 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.835866 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.835867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.835867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.836023 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:17.836081 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:17.836735 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 19:12:17.836805 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2069}
I20260620 19:12:17.837299 23429 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fa81118459ea4cb68d970825cb731ad9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56758
I20260620 19:12:17.837455 23429 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:12:17.837496 23429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:17.838378 23429 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa81118459ea4cb68d970825cb731ad9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } } peers { permanent_uuid: "0d894703052144218e5b46b46e76b16e" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } } peers { permanent_uuid: "c446a54e3a9d40d381c4f46845949d93" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } }
I20260620 19:12:17.838599 23429 leader_election.cc:288] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [CANDIDATE]: Term 1 election: Requested vote from peers 0d894703052144218e5b46b46e76b16e (127.0.203.61:41379), c446a54e3a9d40d381c4f46845949d93 (127.0.203.60:37447)
I20260620 19:12:17.836046 23387 master_runner.cc:186] Error getting master registration for 127.0.203.61:41379: OK, instance_id { permanent_uuid: "0d894703052144218e5b46b46e76b16e" instance_seqno: 1781982737633017 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:12:17.838927 23387 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:12:17.840360 23605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93: Bootstrap starting.
I20260620 19:12:17.841104 23605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:17.841383 23605 log.cc:824] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:17.842298 23605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93: No bootstrap required, opened a new log
I20260620 19:12:17.844285 23605 raft_consensus.cc:348] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa81118459ea4cb68d970825cb731ad9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } } peers { permanent_uuid: "0d894703052144218e5b46b46e76b16e" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } } peers { permanent_uuid: "c446a54e3a9d40d381c4f46845949d93" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } }
I20260620 19:12:17.844434 23605 raft_consensus.cc:729] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c446a54e3a9d40d381c4f46845949d93, State: Initialized, Role: FOLLOWER
I20260620 19:12:17.844518 23605 consensus_queue.cc:260] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa81118459ea4cb68d970825cb731ad9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } } peers { permanent_uuid: "0d894703052144218e5b46b46e76b16e" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } } peers { permanent_uuid: "c446a54e3a9d40d381c4f46845949d93" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } }
I20260620 19:12:17.844827 23605 sys_catalog.cc:564] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:17.845103 23632 sys_catalog.cc:455] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa81118459ea4cb68d970825cb731ad9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } } peers { permanent_uuid: "0d894703052144218e5b46b46e76b16e" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } } peers { permanent_uuid: "c446a54e3a9d40d381c4f46845949d93" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } } }
I20260620 19:12:17.845633 23632 sys_catalog.cc:458] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:12:17.846199 23579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa81118459ea4cb68d970825cb731ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c446a54e3a9d40d381c4f46845949d93"
I20260620 19:12:17.846484 23579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:17.847380 23579 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa81118459ea4cb68d970825cb731ad9 in term 1.
I20260620 19:12:17.847592 23405 leader_election.cc:302] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c446a54e3a9d40d381c4f46845949d93, fa81118459ea4cb68d970825cb731ad9; no voters:
I20260620 19:12:17.847723 23611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:17.847867 23611 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [term 1 LEADER]: Becoming Leader. State: Replica: fa81118459ea4cb68d970825cb731ad9, State: Running, Role: LEADER
I20260620 19:12:17.847932 23458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.839690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56772 (local address 127.0.203.62:39789)
0620 19:12:17.839732 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.839734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.840000 (+ 266us) server_negotiation.cc:407] Connection header received
0620 19:12:17.840106 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.840107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.840118 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.840150 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.840936 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.841563 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.842522 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.842792 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.842835 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.844459 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:17.844460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:17.844461 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:17.844468 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:17.846149 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.846688 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.846689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.846689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.846754 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.846979 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.846980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.846980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.847102 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:17.847235 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:17.847739 (+ 504us) server_negotiation.cc:299] Negotiation successful
0620 19:12:17.847813 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:12:17.847975 23611 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa81118459ea4cb68d970825cb731ad9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } } peers { permanent_uuid: "0d894703052144218e5b46b46e76b16e" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } } peers { permanent_uuid: "c446a54e3a9d40d381c4f46845949d93" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } }
I20260620 19:12:17.848382 23611 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa81118459ea4cb68d970825cb731ad9. Latest consensus state: current_term: 1 leader_uuid: "fa81118459ea4cb68d970825cb731ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa81118459ea4cb68d970825cb731ad9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } } peers { permanent_uuid: "0d894703052144218e5b46b46e76b16e" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } } peers { permanent_uuid: "c446a54e3a9d40d381c4f46845949d93" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } } }
I20260620 19:12:17.848456 23611 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:17.849794 23537 master_runner.cc:186] Error getting master registration for 127.0.203.61:41379: OK, instance_id { permanent_uuid: "0d894703052144218e5b46b46e76b16e" instance_seqno: 1781982737633017 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:12:17.852682 23537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 19:12:17.854007 23643 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:12:17.854096 23643 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:12:17.854276 23646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:17.854535 23646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:17.856298 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.840003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57302 (local address 127.0.203.61:41379)
0620 19:12:17.840037 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.840039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.847682 (+ 7643us) server_negotiation.cc:407] Connection header received
0620 19:12:17.847707 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.847708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.847720 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.847770 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.849453 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.850187 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.850886 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.851190 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.851250 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.852658 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:17.852659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:17.852659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:17.852666 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:17.854547 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.855173 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.855174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.855174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.855249 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.855577 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.855578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.855578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.855732 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:17.855806 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:17.856082 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 19:12:17.856194 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:12:17.856948 23608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.841880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.0.203.60:37447)
0620 19:12:17.841956 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.841958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.846432 (+ 4474us) server_negotiation.cc:407] Connection header received
0620 19:12:17.848699 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.848699 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.848709 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.848741 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.849589 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.850264 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.851805 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.852133 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.852176 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.853597 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:17.853599 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:17.853599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:17.853605 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:17.855348 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.855721 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.855721 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.855722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.855772 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.856025 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.856026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.856027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.856171 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:17.856237 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:17.856749 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 19:12:17.856839 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 19:12:17.857862 23579 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93 [term 1 FOLLOWER]: Refusing update from remote peer fa81118459ea4cb68d970825cb731ad9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:17.858155 23644 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c446a54e3a9d40d381c4f46845949d93" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:17.859959 23632 sys_catalog.cc:455] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa81118459ea4cb68d970825cb731ad9. Latest consensus state: current_term: 1 leader_uuid: "fa81118459ea4cb68d970825cb731ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa81118459ea4cb68d970825cb731ad9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } } peers { permanent_uuid: "0d894703052144218e5b46b46e76b16e" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } } peers { permanent_uuid: "c446a54e3a9d40d381c4f46845949d93" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } } }
I20260620 19:12:17.860049 23632 sys_catalog.cc:458] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:12:17.860975 23611 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa81118459ea4cb68d970825cb731ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa81118459ea4cb68d970825cb731ad9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } } peers { permanent_uuid: "0d894703052144218e5b46b46e76b16e" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } } peers { permanent_uuid: "c446a54e3a9d40d381c4f46845949d93" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } } }
I20260620 19:12:17.861045 23611 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:17.861109 23648 mvcc.cc:205] Tried to move back new op lower bound from 7299001294263468032 to 7299001294226083840. Current Snapshot: MvccSnapshot[applied={T|T < 7299001294263468032}]
I20260620 19:12:17.861526 23632 sys_catalog.cc:455] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa81118459ea4cb68d970825cb731ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa81118459ea4cb68d970825cb731ad9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } } peers { permanent_uuid: "0d894703052144218e5b46b46e76b16e" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } } peers { permanent_uuid: "c446a54e3a9d40d381c4f46845949d93" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } } }
I20260620 19:12:17.861600 23632 sys_catalog.cc:458] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:12:17.861675 23646 catalog_manager.cc:1347] Generated new cluster ID: 829b9f04457c4e3c866a64f72546aef0
I20260620 19:12:17.861709 23646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:17.867872 23530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } attempt: 3
I20260620 19:12:17.868492 23649 mvcc.cc:205] Tried to move back new op lower bound from 7299001294263468032 to 7299001294226083840. Current Snapshot: MvccSnapshot[applied={T|T < 7299001294263468032}]
Jun 20 19:12:17 dist-test-slave-wd30 krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.61@KRBTEST.COM for kudu/127.0.203.60@KRBTEST.COM
I20260620 19:12:17.876185 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.868132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.60:37447 (local address 127.0.0.1:41490)
0620 19:12:17.868182 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:17.868195 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:17.868234 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:17.868427 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:17.868428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:17.868627 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:17.868674 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:17.868677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.869679 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.869681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:17.870458 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:17.870459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.870961 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.870962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:17.871028 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.871595 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:17.871600 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:17.873069 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:17.874983 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:17.874984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:17.874985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:17.875253 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:17.875505 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:17.875506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:17.875506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:17.875575 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:17.875872 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:17.875873 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:17.875929 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 19:12:17.875982 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 19:12:17.876082 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 19:12:17.876693 23608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:17.868273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.0.203.60:37447)
0620 19:12:17.868312 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.868314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.868321 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:12:17.868341 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.868342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.868353 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.868410 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.868819 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.869575 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.870554 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:17.870847 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:17.870896 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:17.873171 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:17.873173 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:17.873174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:17.873181 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:17.874858 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.875358 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.875359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.875359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.875431 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:17.875648 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:17.875649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:17.875650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:17.875785 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:17.875836 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:17.876531 (+ 695us) server_negotiation.cc:299] Negotiation successful
0620 19:12:17.876607 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:12:17.878558 23530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e: Bootstrap starting.
I20260620 19:12:17.879366 23530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:17.879698 23530 log.cc:824] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:17.887887 23530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e: No bootstrap required, opened a new log
I20260620 19:12:17.890071 23530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa81118459ea4cb68d970825cb731ad9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } } peers { permanent_uuid: "0d894703052144218e5b46b46e76b16e" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } } peers { permanent_uuid: "c446a54e3a9d40d381c4f46845949d93" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } }
I20260620 19:12:17.890234 23530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d894703052144218e5b46b46e76b16e, State: Initialized, Role: FOLLOWER
I20260620 19:12:17.890326 23530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa81118459ea4cb68d970825cb731ad9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } } peers { permanent_uuid: "0d894703052144218e5b46b46e76b16e" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } } peers { permanent_uuid: "c446a54e3a9d40d381c4f46845949d93" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } }
I20260620 19:12:17.890712 23530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:17.890810 23503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:17.891948 23503 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e [term 1 FOLLOWER]: Refusing update from remote peer fa81118459ea4cb68d970825cb731ad9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:17.892143 23504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa81118459ea4cb68d970825cb731ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0d894703052144218e5b46b46e76b16e"
I20260620 19:12:17.892923 23504 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa81118459ea4cb68d970825cb731ad9 in term 1.
I20260620 19:12:17.893563 23644 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d894703052144218e5b46b46e76b16e" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:17.890661 23654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa81118459ea4cb68d970825cb731ad9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } } peers { permanent_uuid: "0d894703052144218e5b46b46e76b16e" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } } peers { permanent_uuid: "c446a54e3a9d40d381c4f46845949d93" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } } }
I20260620 19:12:17.894122 23654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:12:17.895776 23654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa81118459ea4cb68d970825cb731ad9. Latest consensus state: current_term: 1 leader_uuid: "fa81118459ea4cb68d970825cb731ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa81118459ea4cb68d970825cb731ad9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } } peers { permanent_uuid: "0d894703052144218e5b46b46e76b16e" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } } peers { permanent_uuid: "c446a54e3a9d40d381c4f46845949d93" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } } }
I20260620 19:12:17.895931 23654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:12:17.896013 23654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa81118459ea4cb68d970825cb731ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa81118459ea4cb68d970825cb731ad9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } } peers { permanent_uuid: "0d894703052144218e5b46b46e76b16e" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } } peers { permanent_uuid: "c446a54e3a9d40d381c4f46845949d93" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } } }
I20260620 19:12:17.896111 23654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:12:17.897306 23666 mvcc.cc:205] Tried to move back new op lower bound from 7299001294263468032 to 7299001294226083840. Current Snapshot: MvccSnapshot[applied={T|T < 7299001294263468032}]
W20260620 19:12:17.897351 23665 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:12:17.899277 23665 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:12:17.910338 23611 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa81118459ea4cb68d970825cb731ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa81118459ea4cb68d970825cb731ad9" member_type: VOTER last_known_addr { host: "127.0.203.62" port: 39789 } } peers { permanent_uuid: "0d894703052144218e5b46b46e76b16e" member_type: VOTER last_known_addr { host: "127.0.203.61" port: 41379 } } peers { permanent_uuid: "c446a54e3a9d40d381c4f46845949d93" member_type: VOTER last_known_addr { host: "127.0.203.60" port: 37447 } } }
I20260620 19:12:17.910480 23611 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:17.915227 23646 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:17.916070 23646 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:17.924305 23646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa81118459ea4cb68d970825cb731ad9: Generated new TSK 0
I20260620 19:12:17.924539 23646 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:12:17.958722 23458 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:17.957234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56784 (local address 127.0.203.62:39789)
0620 19:12:17.957276 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.957279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.958085 (+ 806us) server_negotiation.cc:407] Connection header received
0620 19:12:17.958187 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.958188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.958200 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.958230 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.958654 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56784: BlockingRecv error: recv got EOF from 127.0.0.1:56784 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 19:12:17.959649 23535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:17.957450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.0.203.61:41379)
0620 19:12:17.957486 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.957487 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.959076 (+ 1589us) server_negotiation.cc:407] Connection header received
0620 19:12:17.959205 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.959206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.959216 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.959246 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.959593 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57304: BlockingRecv error: recv got EOF from 127.0.0.1:57304 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 19:12:17.960431 23608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:17.957757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.0.203.60:37447)
0620 19:12:17.957793 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.957795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.959969 (+ 2174us) server_negotiation.cc:407] Connection header received
0620 19:12:17.960057 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.960058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.960067 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.960095 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.960378 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41500: BlockingRecv error: recv got EOF from 127.0.0.1:41500 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 19:12:17.962603 23535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:17.961907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.0.203.61:41379)
0620 19:12:17.961938 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.961940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.962109 (+ 169us) server_negotiation.cc:407] Connection header received
0620 19:12:17.962205 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.962206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.962221 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.962248 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.962562 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57318: BlockingRecv error: recv got EOF from 127.0.0.1:57318 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 19:12:17.965699 23535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:17.964961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.0.203.61:41379)
0620 19:12:17.964994 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.964996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.965226 (+ 230us) server_negotiation.cc:407] Connection header received
0620 19:12:17.965329 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.965330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.965341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.965370 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.965657 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57322: BlockingRecv error: recv got EOF from 127.0.0.1:57322 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 19:12:17.969789 23535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:17.969022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57336 (local address 127.0.203.61:41379)
0620 19:12:17.969057 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.969059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.969270 (+ 211us) server_negotiation.cc:407] Connection header received
0620 19:12:17.969385 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.969386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.969397 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.969426 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.969740 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57336: BlockingRecv error: recv got EOF from 127.0.0.1:57336 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 19:12:17.976078 23535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:17.975231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57352 (local address 127.0.203.61:41379)
0620 19:12:17.975272 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.975274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.975520 (+ 246us) server_negotiation.cc:407] Connection header received
0620 19:12:17.975617 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.975618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.975629 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.975660 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.976023 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57352: BlockingRecv error: recv got EOF from 127.0.0.1:57352 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 19:12:17.986541 23535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:17.985613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.0.203.61:41379)
0620 19:12:17.985659 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:17.985661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:17.985934 (+ 273us) server_negotiation.cc:407] Connection header received
0620 19:12:17.986035 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:17.986037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:17.986050 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:17.986081 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:17.986478 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57368: BlockingRecv error: recv got EOF from 127.0.0.1:57368 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 19:12:18.005175 23535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:18.004138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.0.203.61:41379)
0620 19:12:18.004185 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:18.004188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:18.004499 (+ 311us) server_negotiation.cc:407] Connection header received
0620 19:12:18.004612 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:18.004613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:18.004626 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:18.004658 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:18.005098 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57378: BlockingRecv error: recv got EOF from 127.0.0.1:57378 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 19:12:18.039601 23535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:18.038689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.0.203.61:41379)
0620 19:12:18.038733 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:18.038735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:18.039004 (+ 269us) server_negotiation.cc:407] Connection header received
0620 19:12:18.039102 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:18.039104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:18.039115 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:18.039166 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:18.039540 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57382: BlockingRecv error: recv got EOF from 127.0.0.1:57382 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 19:12:18.105839 23535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:18.105185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.0.203.61:41379)
0620 19:12:18.105289 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:18.105292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:18.105312 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:12:18.105390 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:18.105391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:18.105401 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:18.105467 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:18.105783 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57388: BlockingRecv error: recv got EOF from 127.0.0.1:57388 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 19:12:18.235893 23535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:18.235096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57390 (local address 127.0.203.61:41379)
0620 19:12:18.235176 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:18.235179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:18.235294 (+ 115us) server_negotiation.cc:407] Connection header received
0620 19:12:18.235400 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:18.235401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:18.235414 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:18.235468 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:18.235822 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57390: BlockingRecv error: recv got EOF from 127.0.0.1:57390 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 19:12:18.493721 23535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:18.493045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57404 (local address 127.0.203.61:41379)
0620 19:12:18.493120 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:18.493122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:18.493131 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:12:18.493157 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:18.493158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:18.493166 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:18.493228 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:18.493630 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57404: BlockingRecv error: recv got EOF from 127.0.0.1:57404 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:12:18.854589 23643 catalog_manager.cc:1259] Loaded cluster ID: 829b9f04457c4e3c866a64f72546aef0
I20260620 19:12:18.854699 23643 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93: loading cluster ID for follower catalog manager: success
I20260620 19:12:18.855683 23643 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93: acquiring CA information for follower catalog manager: success
I20260620 19:12:18.856048 23643 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c446a54e3a9d40d381c4f46845949d93: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:12:18.899798 23665 catalog_manager.cc:1259] Loaded cluster ID: 829b9f04457c4e3c866a64f72546aef0
I20260620 19:12:18.899884 23665 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e: loading cluster ID for follower catalog manager: success
I20260620 19:12:18.900736 23665 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e: acquiring CA information for follower catalog manager: success
I20260620 19:12:18.901081 23665 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0d894703052144218e5b46b46e76b16e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:12:19.010282 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:19.006996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57406 (local address 127.0.203.61:41379)
0620 19:12:19.007196 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:19.007199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:19.007323 (+ 124us) server_negotiation.cc:407] Connection header received
0620 19:12:19.007397 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:19.007398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:19.007467 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:19.007577 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:19.007735 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:19.008264 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:19.009347 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:19.009577 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:19.009656 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:19.009897 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:19.010019 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:19.010114 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:19.010137 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 19:12:19.010163 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":57,"threads_started":1}
I20260620 19:12:19.015175 23742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:19.012304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41508 (local address 127.0.203.60:37447)
0620 19:12:19.012435 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:19.012438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:19.012472 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:12:19.012595 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:19.012596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:19.012658 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:19.012778 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:19.012908 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:19.013418 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:19.014198 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:19.014493 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:19.014579 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:19.014763 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:19.014895 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:19.014981 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:19.015022 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 19:12:19.015049 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":54,"threads_started":1}
I20260620 19:12:19.019667 23742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:19.017166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41516 (local address 127.0.203.60:37447)
0620 19:12:19.017234 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:19.017236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:19.017278 (+ 42us) server_negotiation.cc:407] Connection header received
0620 19:12:19.017346 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:19.017347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:19.017357 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:19.017405 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:19.017573 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:19.018060 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:19.018905 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:19.019167 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:19.019230 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:19.019376 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:19.019410 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:19.019466 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:19.019531 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 19:12:19.019556 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:12:19.019951 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:19.016916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57422 (local address 127.0.203.61:41379)
0620 19:12:19.016968 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:19.016970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:19.017200 (+ 230us) server_negotiation.cc:407] Connection header received
0620 19:12:19.017249 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:19.017251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:19.017265 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:19.017316 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:19.017486 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:19.018017 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:19.019091 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:19.019427 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:19.019488 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:19.019664 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:19.019699 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:19.019728 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:19.019857 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 19:12:19.019887 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 19:12:19.020979 23749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:19.016917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56798 (local address 127.0.203.62:39789)
0620 19:12:19.017030 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:19.017032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:19.017258 (+ 226us) server_negotiation.cc:407] Connection header received
0620 19:12:19.017421 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:19.017422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:19.017479 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:19.017580 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:19.018324 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:19.019081 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:19.020133 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:19.020409 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:19.020478 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:19.020663 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:19.020759 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:19.020806 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:19.020898 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 19:12:19.020916 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260620 19:12:19.022644 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 23387
I20260620 19:12:19.028383 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 23462
I20260620 19:12:19.033057 812 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeuOXoM/build/release/bin/kudu with pid 23537
2026-06-20T19:12:19Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2863 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5648 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (270501 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST